DEV Community: Khaled Hosseini

Data structures and algorithms in depth: C++, Swift, Python, Java, C#, JavaScript.

Khaled Hosseini — Wed, 30 Aug 2023 03:05:59 +0000

In programming languages, predefined data structures are widely used, but their names and how they work can differ from one language to another. Despite these differences, the core ideas behind these structures stay the same. In this article, we will take a deep dive into data structures, looking at how they are set up in various programming languages. To make understanding easier, we will include short animations to help you grasp these important concepts. This article marks the beginning of a series, where we delve into a range of essential topics. The series will cover the following main subjects:

Linear data structures
Non-linear data structures
Algorithms

For updated versions, visit github

●  Memory
      ⚬  Physical layer
      ⚬  Virtual layer
            ◾  Location
            ◾  Arrangement
●  Algorithms
      ⚬  Fundamental operations
      ⚬  Fundamental Algorithms
            ◾  Sorting algorithms
            ◾  Searching algorithms
      ⚬  Algorithm design techniques
●  Data structures
      ⚬  Contiguous Memory data structures
      ⚬  Discontiguous Memory data structures
      ⚬  Combination of CM and DCM
      ⚬  Linear data structures
            ◾  Array
            ◾  DynamicArray
            ◾  RingBuffer
            ◾  LinkedList
            ◾  FreeList
            ◾  DoubleLinkedList
            ◾  CircularLinkedList
            ◾  CircularDoubleLinkedList
            ◾  Stack
                  ◽  Stack via DynamicArray
                  ◽  Stack via LinkedList
                  ◽  Stack via deque
            ◾  Queue
                  ◽  Queue via DoubleLinkedList
                  ◽  Queue via RingBuffer
                  ◽  Queue via Double Stack
                  ◽  Deque as Queue
            ◾  Deque
                  ◽  Deque via DoubleLinkedList
                  ◽  Deque via Array
            ◾  PriorityQueue
                  ◽  PriorityQueue via DynamicArray
                  ◽  PriorityQueue via LinkedList
                  ◽  PriorityQueue via Deque
                  ◽  PriorityQueue via BinaryHeap
            ◾  Associative collections
                  ◽  UnorderedMap or HashTable
                  ◽  OrderedMap via HashTable and LinkedList
                  ◽  OrderedMap via HashTable and DynamicArray
                  ◽  SortedMap via Self Balancing Tree
            ◾  Set
                  ◽  UnorderedSet
                  ◽  OrderedSet via HashTable and LinkedList
                  ◽  SortedSet via Self Balancing Tree
●  standard library data structures
      ⚬  C++
      ⚬  Swift
      ⚬  CSharp
      ⚬  Python
      ⚬  Java
●  Non-Linear data structures
●  Algorithms

Suppose you have a collection of data blocks, denoted as |A|, |B|, |C|, |D|, |E|, and so on. Your objective is to solve a problem by employing an algorithm that processes this data and produces a result. Irrespective of the specific problem or algorithm you opt for, there are certain steps that need to be followed:

Allocate some space (memory) to your data.
Arrange your data-blocks in the allocated space and create a logical relationship(Implicit or explicit) among them (Specify a Data Structure).
Doing Some operations on the data-blocks(Algorithm): These operations may include:
- Read data-blocks
- Write data-blocks

Data structures and algorithms are closely intertwined concepts. Certain algorithms exhibit better efficiency when used with specific data structures, and conversely, certain data structures offer advantageous arrangements that enhance the efficiency of particular algorithms. To efficiently solve problems, it is crucial to design both efficient algorithms and appropriate data structures. Designing these efficient solutions necessitates a solid understanding of the fundamentals and analysis techniques involved.

Memory

Physical layer

The physical layer of a computer system is responsible for the actual storage and retrieval of data in electronic or magnetic form. Memory in the physical layer is organized hierarchically, with different types and levels of memory. Types of Memory in the Physical Layer:

Registers
Cache Memory
Main Memory (Random Access Memory - RAM)
Secondary memories: (HDD,SSD,...)

Physical memory is invisible to programs in virtual memory systems and as a programmer you're not required to reason about it.

Memory: Virtual layer

Where and how memory is being allocated in virtual layer?

Location

Stack: Fast allocation. Faster access.
1. Moving just an integer pointer allocates/de-allocates memory.
Heap: Slow Allocation. Slower access.
1. Search heap.
2. Sync heap for other threads.
3. Allocate memory.

Arrangement

Contiguous: Bulk allocation in continuous memory block. (faster access).
Discontiguous: Dynamic allocation in separated memory blocks.(slower access).

Algorithms

At the core of all algorithms are the fundamental operations that involve accessing and potentially mutating the data blocks, regardless of how we arrange our data-blocks in memory and what are the logical connections between them. At this level, all algorithms can be reduced to one or some of the following operations.

Fundamental operations

read
- accessDataBySequence() (Either forward or backward)
- getIndexingInformation(): getStartIndex(), getEndIndex(), getNextIndex(forIndex), getPreviousIndex(forIndex)
- accessDataAtRandomIndex(:): For Random access, time complexity should be of order O(1).
- accessDataAtFront()
- accessDataAtBack()
write
- insertDataAtRandomIndex(:)
- insertDataAtFront()
- insertDataAtBack()
- removeDataAtRandomIndex(:)
- removeDataAtFront()
- removeDataAtBack()
- updateDataAtRandomIndex(:)
- updateDataAtFront()
- updateDataAtBack()

For example, Linear search algorithm uses accessDataBySequence and compare each item with a specified value to find the answer while Binary search algorithm needs accessDataAtRandomIndex operation.

A note on Random Access: In the context of data structures, random access refers to the ability to instantly access a specific location. With Array, for instance, if you select a random index, the Array data structure can immediately provide you with the address of that index. However, if you attempt to access a random index in a LinkedList, the data structure cannot instantaneously provide the address. Instead, it must iterate from the beginning (starting from head) until it reaches the desired index. Consequently, LinkedLists are considered to have a time complexity of O(n) (Upper bound) for random access operation. Most algorithms require O(1) random access, and languages such as Java have introduced a marker interface(with no methods) called RandomAccess. This interface serves as a reminder that certain algorithms rely on random access. To ensure that these algorithms perform efficiently with your data structure, it is necessary to make it compatible with random access. The Swift equivalent is a marker protocol RandomAccessCollection.

Fundamental Algorithms

Fundamental operations form the building blocks upon which algorithms are constructed. Conversely, certain algorithms play fundamental rules for other algorithms. Take, for instance, the impact of input data order on the time efficiency of algorithms. Sorting the data beforehand can greatly simplify our lives, as it has a significant positive effect on the efficiency of numerous algorithms. Sorting can be accomplished through two methods. The first method involves utilizing a sorting algorithm to arrange an unsorted collection. The second method involves utilizing specific data structures, such as binary search trees, that facilitate the sorting of data through amortization.

Sorting algorithms

All sort algorithms need getIndexingInformation, accessDataAtRandomIndex(:) operations. Also items must be comparable (unless for non-comparison algorithms).

In-place sorting algorithms: They need updateDataAtRandomIndex(:) operation.
1. Bubble sort
2. Selection sor
3. Insertion sort
4. Heap sort
5. Quick sort
Not In-Place Sorting Algorithms:
1. Merge sort
2. Radix sort (non-comparison)
3. Bucket sort (non-comparison)

Searching algorithms

Linear search: needs accessDataBySequence()
Binary search: needs accessDataAtRandomIndex(:) with O(1)

Algorithm design techniques

Divide and conquer
Recursion
Randomized algorithms: Input MUST be RANDOM.
Dynamic programming
Greedy algorithms

In a next article, I will return to the algorithms.

Data structures

Each data structure has the following characteristics:

Virtual layer Memory management at the virtual layer.
Logical connection between data-blocks, either implicit or explicit.
- Implicit: In an Array data-blocks have no direct connection, but implicitly they are arranged in a specific order contiguously in memory.
- Explicit: In LinkedList the blocks may not be stored contiguously in memory, but each node has the connection information to some other nodes.
Rules for applying basic operations.
Provides basic read and write operations with a space/time complexity. The space/time complexities for data structures for basic operations can easily be analyzed using the following concepts: Contiguous Memory data structures and Discontiguous Memory data structures

Contiguous Memory data structures

Init with fixed size. size stays fixed.
Address of each block can be calculated via: start + k * blocksize. Random access time complexity is O(1)
Bulk memory allocation
Same size memory blocks (Same type)
Base data Structure example: Array

Discontiguous Memory data structures

This arrangement is a special kind of Graph (We can represent graphs using it).
Each block contains the address of next block.
Time complexity for random access operation is O(n)
Dynamic memory allocation
Memory block sizes can be different (Different types).
Base data structure example: LinkedList

Combination of CM and DCM

A contiguous-memory array of pointers to contiguous-memory or discontiguous-memory collection of objects.
Time complexity for random access operations is O(1) (via array of pointers) but accessing objects in non-continuous memory have a little overhead.
Bulk memory allocation for address (pointer) array, dynamic memory allocation for objects.
Objects can have different memory sizes (different types).
Base data structure example: An array of referenced objects in most programming languages.

Linear data structures

By employing one or a combination of the aforementioned concepts, basic data structures can be implemented, serving as the foundation for more intricate data structures. Additionally, the space and time complexities, as well as memory costs, can be readily analyzed by leveraging the complexities and costs associated with these fundamental concepts.

Array

In Programming languages, Arrays are built-in types. Array of pointers (or array of reference types) acts like Combination of CM and DCM. For primitive types (or value types like Int, enum, struct in C#,Swift,...) if stored in stack, the behavior is like Contiguous Memory data structures. But if the primitives get boxed and be allocated in the heap, the behavior is like Combination of CM and DCM.

Basic operations time complexity: Same as Contiguous Memory data structures
Good:
- accessAtRandomIndex, insertAtBack, removeAtBack operations.
- Bulk memory allocation (fast).
- Contiguous memory. Fast access.
- If used with primitive types (Value types), no dynamic memory allocation cost.
Not good:
- insertAtFront, insertAtMiddle, removeAtFront, removeAtMiddle Operations.
- Fixed size.
Programming Languages implementations:
- CPP: Array size is compile-time constant.
- Swift: Arrays in swift are dynamic.
- Python: Python array size is compile-time constant.
- Java: array size is compile-time constant.
- C#: Array size is compile-time constant.
- JavaScript: Arrays in Javascript are dynamic.

DynamicArray

Similar to array, but can grow at runtime. DynamicArray of pointers (or DynamicArray of reference types) acts like Combination of CM and DCM. For primitive types (or value types like Int, enum, struct in C#,Swift,...) the behavior is like Contiguous Memory data structures. Steps for resizing:

allocate new array with new size
copy the old array values to the new array
delete the old array

Basic operations time complexity: Same as Contiguous Memory data structures
Good:
- accessAtRandomIndex, insertAtBack, removeAtBack operations.
- Bulk memory allocation (fast).
- If used with primitive types (Value types), no dynamic memory allocation cost.
Not good:
- insertAtFront, insertAtMiddle, removeAtFront, removeAtMiddle Operations.
- New memory allocations and copy cost when capacity is full.
- Has unused memory allocation based on growth strategy. For example in Swift programming language, each time an array capacity is full, it double the capacity of the array.
Programming Languages implementations:
- CPP: Vector.
- Swift: contiguousarray and array are dynamic. When capacity is full, the size gets doubled.
- Python: list is a dynamic array of pointers to other objects. The behavior is always like Combination of CM and DCM. UserList is a wrapper class that allows you to create your own list-like objects by inheriting from UserList and implementing certain methods. It provides a convenient way to create custom list-like classes without directly subclassing the built-in list class.
- Java: ArrayList and Vector are dynamic and the difference is that vector is thread-safe.
- C#: ArrayList and List are dynamic arrays. The difference is that ArrayList is non-generic and can store elements of any while List<T> is a generic class that provides type-safe collections.
- JavaScript: When it comes to Javascript, things are a little bit different. Array is dynamic and you can add multiple types to it. As Array is an Object and Objects in javascript are HashTables, you can access indices of array using string of indices too! Depending on the type of the values, the behavior of Javascript array is different.
- In V8 Engine when Array only contains a single primitive type (like integer, float, ...) it’ll be backed by a C++ array of that type and the behavior is like Contiguous Memory data structures.
- When Array contains more than one of primitive types, the array will be backed by a C++ array of the bigger one and the behavior is the same as above.
- If the array contains only objects, or a mixture of numbers and objects, it’ll backed by an array of pointers (primitive types will be boxed inside objects). The behavior is like Combination of CM and DCM.
- When you have a sparse array (WHY?) If it is not too spare, it’ll still be backed by an array, with empty array indices replaced with a ‘hole’ value. If an array is very sparse, it’ll no longer be backed by an array in memory. Instead, it will be backed by a dictionary/hashtable (The key is typically stored as a string representation of the index, and the value is the element itself).

RingBuffer

A ring buffer is a specialized data structure implemented using an array. It is a static-sized buffer where read and write operations occur through two distinct pointers that iterate through the array in a circular manner.

Basic operations time complexity: Same as Array with the following improvement:
- insertAtFront is O(1)
- removeAtFront is O(1)
Good:
- accessAtRandomIndex, insert operation.
- Bulk memory allocation (fast).
- If used with primitive types (Value types), no dynamic memory allocation cost.
- As it is fixed-size, we can map it to virtual memory layer memory page to make it super fast.
Not good:
- Fixed size.
- Write operations may fail if the frequency of writes exceeds the frequency of reads.
Programming Languages implementations:
- CPP: Has no built-in implementation for LinkedList. Here is an implementation.
- Swift: Has no built-in implementation for LinkedList. Here is an implementation.
- Python: Has no built-in implementation for LinkedList. Here is an implementation.
- Java: Has no built-in implementation for LinkedList. Here is an implementation.
- C#: Has no built-in implementation for LinkedList. Here is an implementation.
- JavaScript: Has no built-in implementation for LinkedList. Here is an implementation.

LinkedList

Basic operations time complexity: Same as Discontiguous Memory data structures with one improvement.
- insertAtBack() becomes O(1) because we keep track of tail.
- removeAtBack() stays O(n) because we have to iterate from head to index n-1 to remove n.
Good:
- insertAtFront, removeAtFront, insertAtBack operations.
Not good:
- accessAtRandomIndex, removeAtBack, insertAtMiddle, removeAtMiddle Operations.
- Dynamic memory allocation (slow).
Programming Languages implementations:
- CPP: forward_list.
- Swift: Has no built-in implementation for LinkedList. An implementation can be found here.
- Python: Has no built-in implementation for LinkedList. An implementation can be found here.
- Java: LinkedList is DoubleLinkedList.
- C#: LinkedList is a DoubleLinkedList.
- JavaScript: Has no built-in implementation for LinkedList. an implementation can be found here.

FreeList

As you have noticed, one of the Not Goods of a LinkedList data structure is dynamic memory allocation. It means, whenever you need a new node, you have to create a new one dynamically using new keyword. Dynamic memory allocation is a heavy task. One way of resolving this issue is to use FreeLists. FreeLists can be thought of as a reservoir for the LinkedList nodes. One approach is to initialize a FreeList with a sequence of nodes and whenever you need a Node for your LinkedList, you get one from the FreeList instance and when you remove a Node from the LinkedList, you will not free the memory, but return it to the FreeList reservoir to be used again later. Another approach is the following implementation for LinkedListNode with a private static freelist. In this implementation, the freelist is not initialized with an initial size but it grows as the new nodes are added.

class LinkListNode<E> {      // Singly linked list node with freelist support
    // Extensions to support freelists
    private static LinkListNode freelist = null;                  // Freelist for the class

    private E value;       // Value for this node
    private LinkListNode<E> next;    // Point to next node in list
    // Constructors
    LinkList(E it, LinkListNode<E> inn) { value = it; next = inn; }
    LinkList(LinkListNode<E> inn) { value = null; next = inn; }

    E element() { return value; }                        // Return the value
    E setElement(E it) { return value = it; }            // Set element value
    LinkListNode<E> next() { return next; }                     // Return next link
    LinkListNode<E> setNext(LinkListNode<E> inn) { return next = inn; } // Set next link

    // Return a new link, from freelist if possible
    static <E> LinkListNode<E> get(E it, LinkListNode<E> inn) {
      if (freelist == null) {
        return new LinkListNode<E>(it, inn);                 // Get from "new"
      }
      LinkListNode<E> temp = freelist;                       // Get from freelist
      freelist = freelist.next();
      temp.setElement(it);
      temp.setNext(inn);
      return temp;
    }

    // Return a link node to the freelist
    void release() {
      value = null;   // Drop reference to the element
      next = freelist;
      freelist = this;
    }
  }

DoubleLinkedList

Basic operations time complexity: Same as Discontiguous Memory data structures with two improvements:
- insertAtBack() becomes O(1).
- removeAtBack() becomes O(1). Now we have access to n-1 from n and we can remove the pointer to n from n-1.
Good:
- insertAtFront, removeAtFront, insertAtBack, removeAtBack operations.
Not good:
- accessAtRandomIndex, insertAtMiddle Operations.
- Dynamic memory allocation (slow).
- High overhead of extra storage for the forward and back reference.
Programming Languages implementations:
- CPP: list is doubly linkedList.
- Swift: Has no built-in implementation for DoubleLinkedList. An implementation can be found here.
- Python: Has no built-in implementation for DoubleLinkedList. an implementation can be found here.
- Java: LinkedList is DoubleLinkedList.
- C#: LinkedList is DoubleLinkedList.
- JavaScript: Has no built-in implementation for DoubleLinkedList. An implementation can be found here.

CircularLinkedList

Basic operations time complexity: Same as LinkedList with some more capabilities.
- We can traverse to a previous node
- We can traverse in loop.

CircularDoubleLinkedList

Basic operations time complexity: Same as DoubleLinkedList with some more capabilities.
- We can traverse to a previous node
- We can traverse in loop in both direction.

Stack

Stack is a Last-In-First-Out(LIFO) data structure. Any data structure that is Good at insert/remove from one of the ends can be used as a container for Stack. Based on this, stacks can be implemented using DynamicArray (Good at add/remove from the back), LinkedList (Good at add/remove from front), DoubleLinkedList(Good at add/remove from both front and back) and Deque. Each implementation inherits Good and Not Good of the container data structure.

Stack via DynamicArray

Basic operations time complexity: Same as DynamicArray:
Methods:
- push(): insertAtBack on array container.
- pop: removeAtBack on array container.
Good:
- push() and pop() are O(1) operations.
- Bulk memory allocation for pointers.
- If used with primitive types (value types), no dynamic memory allocation cost.
Not good:
- New memory allocations and copy cost when internal array capacity is full.
- Has unused memory allocation based on growth strategy of the pointer array.
Programming Languages implementations:
- CPP: Stack. In CPP vector, deque and list(DoubleLinkedList) can be used as container for Stack.
- Swift: Has no Stack in standard library. an implementation can be found here.
- Python: Has no built-in Stack in standard library but list can be used as stack in python. An implementation can be found here.
- Java: Stack is implemented with dynamic array.
- C#: Stack is implemented with dynamic array as the container.
- JavaScript: has no built-in stack data structure. an implementation can be found here.

Stack via LinkedList

Basic operations time complexity: Same as LinkedList. We use Head of LinkedList to insert/remove.
Methods:
- push(): insertAtFront on LinkedList container.
- pop: removeAtFront on LinkedList container.
Good:
- push() and pop() are O(1) operations.
Not good:
- accessAtRandomIndex is O(n).
- Dynamic memory allocation (slow).

Stack via deque

Deque data structure can be implemented using Deque via DoubleLinkedList or Deque via Array. The Deque can serve as a container for a Stack due to its behavior. C++ default container for Stack is deque.

Queue

Queue data structure is First-In-First-Out. Every data structure that is Good at addAtFront and removeAtBack or vice versa can be used as a container for Queue data structure. DoubleLinkedList(Good at add/remove at both ends) can be used as the containers for Queue data structure. Also RingBuffer can be used for fixed size queues. DynamicArray: is not a good container for queue data structure because of O(n) for insert operation. We can amortize this complexity using Queue via Double Stack (Stack via DynamicArray). Another approach is storing contents in multiple smaller arrays, allocating additional arrays at the beginning or end as needed. Indexing is implemented by keeping a dynamic array or a LinkedList containing pointers to each of the smaller arrays. In this case, the cost of inserting reduced from O(n) to the O(small_array.length). This approach is used for deque.

Queue via DoubleLinkedList

Basic operations time complexity: DoubleLinkedList
Methods:
- enqueue(): insertAtFront on DoubleLinkedList container.
- dequeue(): removeAtBack on DoubleLinkedList container.
Good:
- enqueue() and dequeue() are O(1) operations.
Not good:
- accessAtRandomIndex operation.
- Extra memory for forward/backward pointers.
- Dynamic memory allocation (slow).
Programming Languages implementations:
- CPP: queue in cpp can has deque or list (DoubleLinkedList) as the container. the default container is deque.
- Swift: Has no built-in implementation for Queue. An implementation can be found here.
- Python: Has no built-in implementation for Queue but list can be used as queue in python. An implementation can be found here.
- Java: LinkedList and ArrayDeque have implemented Queue interface.
- C#:Queue in c# uses circular buffer array.
- JavaScript: an implementation can be found here.

Queue via RingBuffer

Basic operations time complexity: RingBuffer
Methods:
- enqueue(): insertAtRandomIndex on Array container.
- dequeue(): accessAtRandomIndex on Array container.
Good:
- enqueue() and dequeue() are O(1) operations.
- If used for primitive types (value types), No dynamic allocation.
Not good:
- Fixed size, enqueue() may fail.
Programming Languages implementations:
- C#: Queue in c# uses circular buffer array.

Queue via Double Stack

If we use DynamicArray as container for our queue, the dequeue() time complexity would be O(n) (Adding items to start of an array is an O(n) operation ). But we can amortize this complexity to O(1) using two stacks. LeftStack for enqueue() and the RightStack for dequeue(). Each time the LeftStack is empty, copy the RightStack contents to the LeftStack. This operation guarantees First-In-First-Out for the queue.

Basic operations time complexity: Similar to Stack via DynamicArray.
Methods:
- enqueue(): insertAtBack on left Array container (the enqueue stack).
- dequeue(): removeAtBack on right Array container (the dequeue stack).
Good:
- enqueue() and dequeue() are O(1) operations.
- If used for primitive types (value types), No dynamic allocation.
Not good:
- New memory allocations and copy cost when capacity is full.
- Has unused memory allocation based on growth strategy.

Deque as Queue

Deque (Double-Ended Queue) can be used as Queue.

Deque

Deque (Double-Ended Queue) are a type of Queue that enqueue() and dequeue() can happen at both ends. Every data structure that is Good at insert/remove from both ends can be used as a container for Deque data structure. The only data structure that fullfil this requirement is DoubleLinkedList. Array is not a good data structure for implementing Deque data structure directly. However we can use some tricks to use Array as a container for Deque data structure. See Deque via Array.

Deque via DoubleLinkedList

Implementing a Deque via DoubleLinkedList is straightforward as this data structure has O(1) for insertAtFront/removeAtFront and insertAtBack/removeAtBack operations.

Methods:
- pushBack(): insertAtBack of the DoubleLinkedList container.
- pushFront(): insertAtFront of the DoubleLinkedList container.
- popBack(): removeAtBack of the DoubleLinkedList container.
- popFront(): removeAtFront of the DoubleLinkedList container.
Good:
- Easy implementation
Not Good:
- Random access operation.
- Dynamic memory allocation (slow).
- High overhead of extra storage for the forward and back references.
Programming Languages implementations:
- Python: deque uses DoubleLinkedList internally.

Deque via Array

As it was the case for Queue data structure, Array cannot be used as a container for Deque data structure directly because insertAtFront/removeAtFront operations are not O(1) for Arrays. We can use one of the following techniques to use Array as a container:

Using a special RingBuffer.
Using an Array and allocating deque contents from the center of the underlying array, and resizing the underlying array when either end is reached.
Storing contents in multiple smaller arrays, allocating additional arrays at the beginning or end as needed. Indexing is implemented by keeping a dynamic array containing pointers to each of the smaller arrays. In this case, the cost of resizing the array in step 2 is eliminated but different small arrays are not allocated contiguously in memory.

Good:
- Random Access operation
Not Good
- More complex implementation
- Need for array resize when filled
Programming Languages implementations:
- CPP: Deque uses approach 3 of above mentioned tricks to use Array as container for Deque. In this approach data is stored in smaller arrays and these arrays are linked using a doubleLinkedList or another array.
- Swift: Has no built-in implementation for LinkedList. An implementation can be found here.
- Python: deque uses DoubleLinkedList internally.
- Java: ArrayDeque is implemented using technique 1 of above mentioned tricks (Circular buffer).
- C#:Deque is implemented using technique 1 of above mentioned tricks (Circular buffer).
- JavaScript: An implementation can be found here.

PriorityQueue

PriorityQueue is the same as Queue with one difference. The dequeue operation is not for the first item that has been inserted. Instead the dequeue item is selected based on a priority criteria and the item may be at the front, the middle or the end of the collection. Any data structure that is Good at inserting at one of the ends can be used as a container for PriorityQueue. As finding the item to be dequeued includes a searching phase, for linear data structures as the container for PriorityQueue the time complexity of dequeue operation is O(n). In case of Heap data structure as the container, the time complexity reduces to O(log(n)) due to internal structure of the Heap.

PriorityQueue via DynamicArray

Methods:
- enqueue(): insertAtBack on Array container.
- dequeue(): iterate and then removeAtMiddle on Array container. Time complexity is O(n).
Good:
- enqueue() is O(1) operation.
- If used for primitive types (value types), No dynamic allocation.
Not good:
- dequeue() operation is O(n).
- New memory allocations and copy cost when capacity is full.
- Has unused memory allocation based on growth strategy.
Programming Languages implementations:
- CPP: priority_queue is using deque as a container by default. vector also can be used.

PriorityQueue via LinkedList

Methods:
- enqueue(): insertAtFront on LinkedList container.
- dequeue(): iterate and then removeAtMiddle on LinkedList container. Time complexity is O(n).
Good:
- enqueue() is O(1) operation.
Not good:
- dequeue() operation is O(n).
- Dynamic memory allocation (slow).

PriorityQueue via Deque

Deque data structure can be implemented using either Deque via DoubleLinkedList or Deque via Array and PriorityQueue can use it as a container.

PriorityQueue via BinaryHeap

Methods:
- enqueue(): insert on BinaryHeap container.
- dequeue(): delete on BinaryHeap container.
Good:
- dequeue() is O(log(n)) operation.
Not good:
- enqueue is O(log(n)) operation. In PriorityQueue via DynamicArray and PriorityQueue via LinkedList this operation is O(1).
Programming Languages implementations:
- Java: PriorityQueue uses binary heap as internal data structure.
- C#:PriorityQueue uses binary heap as internal data structure.

Associative collections

An associative collection is an abstract data type that stores a collection of (key, value) pairs, ensuring that each possible key appears at most once in the collection. However, there is no standardized naming convention for these types of data structures, leading to varying terminology across different programming languages, which can cause confusion. Some alternative names for associative collections include associative array, map, symbol table, or dictionary. See here.

UnorderedMap or HashTable

Other name is HashTable. The main idea behind a Hashtable is to use a hashing function to map keys to specific buckets or slots in an array. Each bucket can store one or more key-value pairs. Hash functions can occasionally generate the same index for different keys, resulting in a collision. To handle collisions efficiently, Hashtable data structures employ various strategies:

Each bucket in the array is a LinkedList of key-value pairs.
Open addressing
Resizing the Array.

‌For most data structures, a linear search is an O(n) or O(log(n)) operation. HashTable is a data structure with an amortized O(1) time complexity for searching. Length of arrays in a HashTable is a prime number.

Good:
- O(1) for search operation.
Not Good:
- Collection has no order. No Random access.
- If LinkedList used for collision handling: Worst-case for search can be O(n) (All nodes collide). Average-case is not O(1).
Programming Languages implementations:
- CPP: unordered_map is an unordered collection created using HashTable. Another version is unordered_multimap that allows for repetitive keys. in the unordered_map version the keys are unique.
- Swift: Dictionary is an unordered collection created using HashTable. The keys are unique.
- Python: dict is an unordered map created using HashTable. Also Counter is a dictionary specific to counting of values (the key is the item you put in the dictionary and the value is a counter. on each insert, if the value exists, 1 is added to the count). UserDict is a wrapper class that allows you to create your own dictionary-like objects by inheriting from UserDict and implementing certain methods. It provides a convenient way to create custom dictionary-like classes without directly subclassing the built-in dict class. mappingproxy object provides read-only access to the original dictionary's data.
- Java: HashTable is unordered, thread-safe. HashMap is unordered map created using HashTable.
- C#: Dictionary is an unordered map created using HashTable. ListDictionary uses a combination of array (for keys) and LinkedList (for values). Operations are all O(n) and it MUST be used for small collections (Less than 10 items).
- JavaScript: Map is an unordered map.

OrderedMap via HashTable and LinkedList

A collection of key-value pairs. While the order of the insertion is preserved, the collection is not sorted.

Good:
- Order of the insertion is preserved. (Unlike SortedMap, the keys are not sorted.)
- accessDataBySequence is possible.
Not Good:
- No random access with O(1) because of LinkedList.
- High overhead of extra storage for the forward and back reference.
Programming Languages implementations:
- Python: OrderedDic is implemented using a combination of a doubly linked list and a dictionary.
- Java: LinkedHashMap. In Java, the LinkedHashMap class uses a combination of a hash table and a doubly linked list as its internal data structure to provide the functionality of a hash map with predictable iteration order.

OrderedMap via HashTable and DynamicArray

A collection of key-value pairs. While the order of the insertion is preserved, the collection is not sorted.

Good:
- Order of the insertion is preserved. (Unlike SortedMap, the keys are not sorted.)
- accessDataBySequence is possible.
- accessDataAtRandomIndex is O(1).
Not Good:
- insert is O(n) because of array.
- remove is O(n) because of array.
Programming Languages implementations:
- C#: OrderedDictionary uses a combination of a HashTable and an ArrayList.

SortedMap via Self Balancing Tree

A collection of key-value pairs which is sorted by the key.

Good:
- Search is O(log(n))
- keys are sorted.
Not Good:
- Random access is not O(1).
- Suitable for small number of data.
Programming Languages implementations:
- CPP: map uses Red-Black Tree for implementation. Another version is multimap which allows duplicate keys. In the map version, keys are unique.
- Swift: Swift has no built-in Ordered map using tree data structure. You can sort the keys of a dictionary to a collection and iterate that collection.
- Python: Swift has no built-in Ordered map using tree data structure.
- Java: TreeMap is implemented using a Red-Black Tree as its internal data structure.
- C#: SortedDictionary is implemented internally using a self-balancing binary search tree called a Red-Black Tree. SortedList uses two separate arrays. one for the keys and the second for the values. As the array for the keys is sorted, when a new item is inserted, the index is found via binary search. The time complexity for inserting is O(n). Binary search is O(log(n))and the items re-arrangement is O(n).
- JavaScript: an implementation can be found here.

Set

UnorderedSet

It is almost exactly like UnorderedMap or HashTable with the distinction that the node has only a key and no value exists. In Java, it is implemented using HashTable and the values for the nodes are set to a fixed value.

Good:
- O(1) for search operation.
Not Good:
- Collection has no order. No Random access.
- If LinkedList used for collision handling: Worst-case for search can be O(n). Average-case is not O(1).
Programming Languages implementations:
- CPP: unordered_set is an unordered collection created using HashTable. Another version is unordered_multiset that allows for duplicate keys. in the unordered_set version the keys are unique.
- Swift: Set is an unordered collection created using HashTable. The keys are unique.
- Python: Set is an unordered set created using HashTable. frozenset is an immutable set.
- Java: HashSet is an unordered set created using HashTable.
- C#: HashSet is an unordered set created using HashTable.
- JavaScript: Set is an unordered set.

OrderedSet via HashTable and LinkedList

It is almost exactly like OrderedMap via HashTable and LinkedList with the distinction that the node has only a key and no value exists. In Java, it is implemented using HashTable and the values for the nodes are set to a fixed value.

Good:
- Order of the insertion is preserved. (Unlike SortedSet, the keys are not sorted.)
Not Good:
- No random access with O(1) because of LinkedList.
Programming Languages implementations:
- Java: LinkedHashSet. In Java, the LinkedHashSet class uses a combination of a hash table and a doubly linked list as its internal data structure to provide the functionality of a hash set with predictable iteration order.

SortedSet via Self Balancing Tree

Good:
- Search is O(log(n))
- keys are sorted.
Not Good:
- Random access is not O(1).
- Suitable for small number of data.
Programming Languages implementations:
- CPP: set uses Red-Black Tree for implementation. Another version is multiset which allows duplicate keys. In the Set version, keys are unique.
- Swift: Swift has no built-in Ordered set. You can sort the keys of a set to a collection and iterate that collection.
- Python: Python has no built-in Ordered set.
- Java: TreeSet is implemented using a Red-Black Tree as its internal data structure.
- C#: SortedSet is implemented internally using a self-balancing binary search tree called a Red-Black Tree.
- JavaScript: An implementation can be found here.

standard library data structures

C++

Swift

Swift source code for collections can be found here.

CSharp

Dotnet source code for collections can be found here.

Python

Source code for python built-in types can be found here. Collection module source code is located here.

Java

Java collections source code is located here.

Non-Linear data structures

Coming soon

Algorithms

Coming soon

Master typescript types in 7 minutes.

Khaled Hosseini — Sat, 05 Aug 2023 19:00:09 +0000

Types in C# and Java

You may be familiar with the types in managed-type languages like Java or C#. In these languages, we have two kinds of types.

primitive types

These types are value-based

These types are predefined by the language

Examples are int, string, float, bool, byte, double,

In c#, you can use typeof operator to get the type of a type and GetType() method to get the type of a variable.

Console.WriteLine(typeof(int)); // System.Int32
Console.WriteLine(typeof(byte)); // System.Byte
Console.WriteLine(typeof(double)); // System.Double
Console.WriteLine(typeof(string)); // System.String
var str = "string";
Console.WriteLine(str.GetType()); // System.String

In java, you can use instanceof operator to check if an object is a type or not

Object d = 0.0;
System.out.println(d instanceof Double);//true

Non-primitive types

These types are generally referenced based (except Enums and struct in c#)

These types are user-defined types and users can define and create them using class, enum, struct, interface, delegate, etc.

In c# you can use typeof operator to get the type of a Type and GetType() method to get the type of a variable.

class Program {
    public static void Main() {
       Console.WriteLine(typeof(Program));//Program
       var program = new Program();
       Console.WriteLine(program.GetType());//Program
   }
}

In Java, you can use instanceof operator.

class Car{
   interface IExample {
         public void print();
   }
   public static void main(String args[]){
     Car c=new Car();
     System.out.println(c instanceof Car);//true

     Object obj = new IExample() {
            public void print() {

            }
        };
     System.out.println(obj instanceof IExample);//true
   }
}

Types in typescript

primitive types

Typescript's primitive types are string, number, boolean, bigint, undefined

These types are value-based

These types are predefined by the language

You can use typeof operator to get the types of values.

    let big1 = 9007199254740991n;
    console.log(typeof(big1)) // "bigint"

    let bool = true
    console.log(typeof bool) // "boolean"

    let undef
    console.log(typeof undef)//"undefined"

You can consider primitive types in Typescript to work similarly to those in Java and C#, but actually they do not! Javascript types have more to offer. You will see in a moment.

Non-primitive types

These types are generally referenced based (except Enums)

These types are user-defined types and users can define and create them using class, type, interface, function, etc.

Do not confuse the general word type which is a programming concept with Typescript's type keyword which is one way of creating types.

You can use typeof operator to get the types of values.

    type Obj = { x: string }
    const obj: Obj = { x: "" }
    console.log(typeof obj) // "object"

    class Class {  constructor(public a: number){} }
    const cls: Class = new Class(1)
    console.log(typeof cls)// "object"

    interface Interface { i: number }
    const obj2: Interface = {  i: 1 }
    console.log(typeof obj2)// "object"

    let nl = null
    console.log(typeof nl)// "object"

As you may have noticed, when it comes to the typeof operator in TypeScript, the results for non-primitive types are all "object". Here, it's important to adjust our understanding of types in TypeScript. In Typescript Instead of thinking of a type as a rigid definition, consider it as a set (Unique values) that encompasses multiple values. This characteristic makes TypeScript types incredibly flexible and powerful. In languages like C# and Java, a variable can only have a single type, and you can create new types by utilizing techniques like inheritance or interface implementation. However, in TypeScript, you not only have access to those same techniques, but you also have a wide range of additional methods to manipulate types like combining them, filtering a type to create a new type, map an existing type to another one and so on. Below I introduce the patterns that you can use to work with types in Typescript and see how they are different from managed-type languages.

Unions of primitive types / values

In languages like C# and java, a variable can only be one type at the same time. For example either a string or number and so on. In Typescript you can use the union of multiple types as a new type or union of multiple values as a new type.

    //union of two primitive types
    type stringOrNumber = string | number;
    let son: stringOrNumber = "text"
    console.log(typeof son) // "string"
    console.log(son) // "text"
    son = 1
    console.log(typeof son) //number
    console.log(son) // 1

    //union of multiple number values
    type DiceNumber = 1 | 2 | 3 | 4 | 5 | 6;
    let diceNumber: DiceNumber = 1
    //let diceNumber: DiceNumber = 10 // error: Type '10' is not assignable to type 'DiceNumber'
    console.log(typeof diceNumber) // "number"
    console.log(diceNumber) // 1

    //template union types
    type Langs = 'en' | 'fr'
    type texts = 'footer' | 'header'
    type combine = `${Langs}_${texts}`
    /*
        type combine = "en_footer" | "en_header" | "fr_footer" | "fr_header"
    */

Extend `object` types

You can extend type, class or interface types to create new types. For classes and interfaces, aside from minor differences, it is what you are also doing in languages like C# or Java.

    type Person = { name: string; age: number;};
    //Create a new type by removing the name
    type PersonWithoutName = Omit<Person,'name'>
    //create new type by adding id to Person type
    type PersonWithId = {id: string; } & Person;
    //create a new type using intersection of two other types
    type Dim1 = {x: number}
    type Dim2 = {y: number, z: number}
    type Dim3 = Dim1 &  Dim2

    interface IPersion {name: string}
    //create new type by extending IPersion
    interface IPersopnWithId extends IPersion { id: number;}
    // Add properties to an existing interface by redeclaring the interface name. (You cannot do this for `type`)
    interface IPersion {age: number}

    class Parent {
        constructor(public a: number) {}
    }
    // Create a new type by extending Parent class
    class Child extends Parent {
        constructor(public b: number, public a: number)
        {
            super(a);
        }
    }

Map an existing type to another by applying a `mapper type`

You can create a new type from an existing type by changing the properties of the old one using a mapper.

    type Artist = {
        id: number;
        name: string;
        bio: string;
    }
    // a mapper that makes all fileds of the old type optional and add a mandatory id filed
    type MyPartialTypeForEdit<Type> = {
    [Property in keyof Type]?: Type[Property];
    } & { id: number };
    // this type contains all fileds from Artist type but optional
    type ArtistOptionalFields = MyPartialTypeForEdit<Artist>;
        /*
        type ArtistOptionalFields = {
            id?: number | undefined;
            name?: string | undefined;
            bio?: string | undefined;
        } & {
            id: number;
        } 
        */

Filter types by applying a `conditional type`

You can create a new type by setting some rules. Then apply this conditional type to a compound type to filter it.

    // If A extends this type-> {legs: 4}, then A has four legs, otherwise never.
    type HasFourLegs<A> = A extends {legs: 4} ? A : never
    type Bird = { legs: 2, wings: number };
    type Dog = { legs: 4, breed: string};
    type Ant = { legs: 12};
    type Lion = { legs: 4};
    type Animals = Bird | Dog | Ant | Lion
    type four_legs = HasFourLegs<Animals> // Dog | Lion

Mix classes using `mixins`

In typescript, like most other programming languages, a class can only inherit from one parent class. But what if we want to have the capabilities of multiple classes in one class (just like C++). Do not worry. Typescript allows you create a new type by mixing classes.

    class Jumpable { jump() { console.log("jump");} }
    class Duckable { duck() {console.log("duck");} }
    //Our mixed class
    class Sprite { x = 0; y = 0; }
    // Then you create an interface which merges
    // the expected mixins with the same name as your base
    interface Sprite extends Jumpable, Duckable {}
    // Apply the mixins into the base class via the JS at runtime
    applyMixins(Sprite, [Jumpable, Duckable]);

    let player = new Sprite();
    player.jump();
    player.duck();

    // This can live anywhere in your codebase:
    function applyMixins(derivedCtor: any, constructors: any[]) {
        constructors.forEach((baseCtor) => {
            Object.getOwnPropertyNames(baseCtor.prototype).forEach((name) => {
            Object.defineProperty(
                derivedCtor.prototype,
                name,
                Object.getOwnPropertyDescriptor(baseCtor.prototype, name) ||
                Object.create(null)
            );
            });
        });
    }

Picking properties from another type

You can use Pick<> to pick certain properties from another type and create a new type.

Type User = {
    id: string,
    name: string,
    age: string,
    email: string,
    phone: number,
    password: string
}

Type LoginUser = Pick<User,'email' | 'password'>

What else you can add to this list?

How TLS / mTLS handshakes Work?

Khaled Hosseini — Tue, 01 Aug 2023 21:04:24 +0000

TLS/SSL (transport layer security)

TLS is a transport layer security protocol (the successor of the Secure Sockets Layer (SSL) protocol) and it secure the network communication in two ways:

Encrypting the traffic against eavesdroppers on the internet
Validating the authenticity of the counterparty

In this protocol, client applications prioritize verifying the identity of the server application. For instance, when entering an address like google.com in the web browser's address bar, it becomes crucial to ensure a secure connection with the legitimate google.com website, avoiding any potential connection to a different or malicious site. Here is a schematic flow of how a typical TLS connection is established:

In TLS (Transport Layer Security), the trust is established through a trusted certificate authority. This authority possesses a public-private key pair. The client applications have access to the public key of this certificate authority.
To secure the server's identity, the server generates its own public-private key pair. It then creates a Certificate Signing Request (CSR) that includes relevant information such as the website's domain name and the server's public key. This CSR is sent to the certificate authority for verification.
Upon receiving the CSR, the certificate authority validates the information provided, signs it using its private key, and issues a certificate for the server. This certificate contains the original information submitted by the server, along with a digital signature. This signature can be verified using the public key of the certificate authority.
These steps are essential prior to initiating a TLS handshake between the client application and the server, ensuring a trusted and secure connection. The next steps are as follows:

The client say hello to the server
Server replies and send it's certificate
The client verifies the signature of the certificate using the CA public key.
If verified, the client generate an ESA key, encrypt it with the public key of the server and send it back to the server. These key can only be read by the server, because it is encrypted with the public key of the server and can only be decrypted using the private key of the server.
Subsequent messages communicated between client and the server will be encrypted with that ESA key. because only the client and the server have this key, only these two can decrypt the messages.

mTLS (mutual transport layer security)

Consider a scenario where both communicating parties desire to authenticate each other's identities. This is where the mTLS (Mutual Transport Layer Security) protocol comes into play, offering a solution. mTLS is a transport layer security protocol (like TLS) and it secure the network communication in two ways:

Encrypting the traffic against eavesdroppers on the internet
Validating the authenticity of the counterparty

In this protocol, the foundation of trust lies once again in a trusted certificate authority. For successful implementation, both communicating parties are required to register their identities with this trusted certificate authority. The handshake process resembles that of TLS, with the distinction that both sides must exchange their certificates with each other.

Play Microservices: Security

Khaled Hosseini — Sun, 30 Jul 2023 08:19:38 +0000

The source code for the project can be found here:

End user: An individual or entity that utilizes and interacts with a computer system or network.
Client app: Software application that sends requests for services or resources from a server.
Server app: Software application that responds to and fulfills client requests by providing services or resources.
Authentication: Verifying the identity of a user, device, or system to ensure they are who they claim to be.
Authorization: Granting or denying access rights and permissions to users, devices, or systems based on their authenticated identity and assigned privileges.
JWT: JSON Web Token; a compact and self-contained data format for securely transmitting information between parties.
Certificates: Digital documents that verify the authenticity of an entity and enable secure communication, often using public key cryptography.
Self-signed certificate: A certificate signed by the entity it belongs to, not by a trusted third-party Certificate Authority (CA).
Hashing: Process of converting data into a fixed-length string of characters through a mathematical algorithm, commonly used for data integrity and password storage.
RSA: A widely used public-key cryptosystem for secure data transmission and encryption/decryption of sensitive information.

security protocols

TLS/SSL (Transport layer security)

TLS is a transport layer security protocol (the successor of the Secure Sockets Layer (SSL) protocol) and it secure the network communication in two ways:

Encrypting the traffic against eavesdroppers on the internet
Validating the authenticity of the counterparty

The client say hello to the server
Server replies and send it's certificate
The client verifies the signature of the certificate using the CA public key.
If verified, the client generate an ESA key, encrypt it with the public key of the server and send it back to the server. These key can only be read by the server, because it is encrypted with the public key of the server and can only be decrypted using the private key of the server.
Subsequent messages communicated between client and the server will be encrypted with that ESA key. because only the client and the server have this key, only these two can decrypt the messages.

mTLS (mutual transport layer security)

Encrypting the traffic against eavesdroppers on the internet
Validating the authenticity of the counterparty

OAuth 2.0

In TLS and mTLS our source of trust is a trusted certificate authority that issues certificates for communication parties. In this case, the party for which the certificate is issued is a company or organization, not an individual. Now imagine cases where our goal is to authenticate individuals or client apps without using the certificate mechanism. (In this case, issuing a certificate for an individual is meaningless). This is where the protocols like oauth 2.0 comes into play. This protocol runs on application layer and depending on use cases, there are several versions of it.

Use case 1: A client app (Not a user) is the owner of the resource.

In this scenario, the client app (Not a user or individual) which is a machine, is the owner of a resource. The identity authentication is done for a client app and not a user. The schematic flow for OAuth 2.0 for this use case is as follows. see Client Credentials Flow:

Notes:
- The identity of the client application is authenticated.
- The client secret and client ID should be kept confidential.
- The client secret and ID are transmitted securely using TLS or mTLS.

Please note that in this scenario, the client app is not in a public access domain and the client secret and the client id should be kept secret(For example a web application that runs on a server, not a mobile application that is downloaded to the user's devices).

Use case 2: User is the owner of a resource and the user request resource from a registered client app in private domain.

In this scenario, User is the owner of a resource. The user must request for access from a trusted client app. In this case, the client app should not be accessible publicly so that its client id and client secret can be kept safe (For example a web application that runs on a server, not a mobile application that is downloaded to the user's devices). The schematic OAuth 2.0 flow for this scenario is as follows. see Authorization Code Flow.

Notes:
- Identity of the client app is authenticated and unauthorized client apps cannot connect to resource server.
- Identity of the user is authenticated
- Client app have no access to user's credentials and this can provide mechanism for 3rd party login like login via google, etc.
- All credentials are transmitted securely using TLS or mTLS.

An implementation commonly used for this scenario is third-party authentication provider login. For example, when you want to add login via google to your web app, this flow is the one you need to go through. See here for more info.

Use case 3: User is the owner of the resource and it requests the resource from any client app.

In this scenario the user is the owner of the resource, but it is not necessary to request access to the resource via a trusted and registered client app. In other words, all client apps are trusted and we do not authenticate them in this scenario (We only authenticate the user). In our sample microservice application, we do the exact thing for authentication of the user. The schematic diagram for this flow can be shown as below. see Resource Owner Password Flow.

Note:
- Only user's identity is authenticated.
- Used for highly trusted client applications.
- Not recommended.

Use case 4: User is the owner of the resource and it requests the resource from a client app in a public domain (Like mobile or desktop app).

In this scenario, The user do the request from an application on public domain (for example a mobile app or a desktop app). In this case, the app cannot hide client id and client secret and therefore we do not authenticate the client application identity. We cannot use Authorization Code Flow in this scenario because If the auth code somehow be stolen (By changing the redirect url for example), it can be exchanged with an access token. That's why Authorization Code Flow with Proof Key for Code Exchange (PKCE) has been introduced.

Notes:
- Only identity of the user is authenticated.
- To prevent hijacking of auth code and exchange it with an access token, code verifier/code challenge mechanism is used. Client app generates a cryptographic random code. This is the code verifier and the client save it in a safe place. Then hash it using an algorithm like s256. The hashed code is called the code challenge. The client app request auth code by sending code challenge. The auth server redirect to login page and after login, it redirects the auth code to the address defined by the client app. If by any means, this auth code revealed to an attacker, it is useless without the code verifier and it cannot be exchanged with an access token.

Security in microservices

When it comes to securing a microservice, there are four fundamental areas to consider:

Access to microservices from outside(known as north-south traffic)
Access between microservices (known as east-west traffic)
Microservice container contents
Deployment environment

1. Access to microservices from outside(known as north-south traffic)

The methods that we can incorporate to secure accessing our microservice application from outside can be summarized as follows:

Enabling TLS

Using API gateways

Using rate limiters

Using OAuth 2.0 protocol to implement authentication and authorization. Depending on your priorities OAuth protocol can be implemented in several ways.

Implementing security at each microservice level: In this approach we follow zero-trust security model which says never trust, always verify. In this approach Auth service is responsible for issuing and signing the tokens. Then all other services do authentication and authorization using the public key of the auth service.

Implementing authentication in api-gateway and do authorization in all services: In this approach, api-gateway do the authentication and then attach authorization headers to the requests. Other services trust the api-gateway and do only the the authorization. This approach can be somehow safe if we use mTLS for communication between services. In this case, Other services can be sure that they are talking to the api-gateway.

Another approach can be doing both authentication and authorization in one service. Aside from security problems that this approach can have, doing authorization for all other services inside one service require a strong coupling between all services and the authorizer service which by nature is against philosophy of microservice architecture.

2. Access between microservices (known as east-west traffic)

The methods that we can use to secure microservices inter-connections is as follows:

Enabling TLS

Enabling mTLs

Implementing Kubernetes networking policies

TLS and mTLS is used for authentication of services in the edge. For example, When using mTLS, Service A can be sure that it is talking to service B and vice versa. Another layer of security that we can add too this layer is to add authorization. For example, When issuing a certificate for service A (In a self-signed certificate), we determine the access and role for that service too. In this case, service B can check both the identity of service A and whether service A has the permission to do some operation or not. Networking policies are a specific Kubernetes tool for implementing east-west security. You can set per-pod criteria that define which other pods are allowed to communicate with the target. This way, Services that have no connection can be separated in the network layer.

3. Microservice container contents

The security of microservices extends beyond network security. What if a malicious code is running inside your container? Or You are using a package with a security hole? using hardened base images or assembling your own from scratch will help to ensure there is nothing dangerous lurking within. Also Automated security testing techniques like DAST, SAST, and IAST can be used to detect possible flaws in your code.

4. Deployment environment

Hardening the environment that hosts your deployment is the final element of microservices security. Basic cloud security hygiene measures (e.g., limiting user privileges and regularly rotating access tokens) are vital, but there are also specific best practices for distributed systems like Kubernetes. The Kubernetes RBAC system should be used to configure access for each user and service account in your cluster. Restricting accounts to the bare minimum privileges their functions demand will also improve your security posture.

Play Microservices: Integration via docker-compose

Khaled Hosseini — Thu, 27 Jul 2023 02:48:31 +0000

This is the 8th part of a series of articles under the name "Play Microservices". Links to other parts:
Part 1: Play Microservices: Bird's eye view
Part 2: Play Microservices: Authentication
Part 3: Play Microservices: Scheduler service
Part 4: Play Microservices: Email service
Part 5: Play Microservices: Report service
Part 6: Play Microservices: Api-gateway service
Part 7: Play Microservices: client-web service
Part 8: You are here
Part 9: Play Microservices: Security

The source code for the project can be found here:

Auth service (plus a database and a cache service)
Job Scheduler service (plus a database, a kafka broker and a database for kafka broker)
Email job executor service
Report service (plus a database, a kafka broker and a database for kafka broker)
An api-gateway service which acts as an entry-point to all other services.
A client-web service

Now it's time to bring everything together. But before, lets have a look at the tools we need.

Tools

The tools required In the host machine:

Docker: Containerization tool
Git

Integration via Docker-compose

If you have followed the previous sessions, you should now have the following folder structure for all our services.

Inside play-microservices folder, create a file named .env and set the contents to:

AUTH_SERVICE_PORT=9001
AUTH_POSTGRES_PORT=5432
AUTH_REDIS_PORT=6379
ZOOKEEPER_PORT=2181
KAFKA1_PORT=9092
SCHEDULER_MONGODB_PORT=27017
SCHEDULER_SERVICE_PORT=9002
EMAIL_SERVICE_PORT = 9003
POSTFIX_PORT=25
REPORT_SERVICE_PORT = 9004
REPORT_MONGODB_PORT=27018
API_GATEWAY_PORT=5010
CLIENT_PORT = 3000

This file make it easier for us to handle the ports used by different services.

Now, let's create a file called docker-compose.yml inside the play-microservices folder. This file will be used to integrate all the services. The contents of the docker-compose file can be found here.

Some notes on the docker-compose.yml file

As depicted in the services diagram, the services exhibit a hierarchical dependency tree. Therefore, inside the docker-compose file, we need to orchestrate the services in such a way that when a service starts, its dependencies are started and running in advance. To achieve this, we use healthcheckof docker compose. for example, for auth-db, which is a postgres service, for the healthcheckpart we have:

    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 10s
      timeout: 1s
      retries: 5

Auth service has dependency to auth-db service. For the depends_on condition we have:

    depends_on:
      auth-db:
        condition: service_healthy

Seting condition to service_healthy guarantee that auth service will not run unless the auth-db is ready and running.

We have other healthcheck definitions for zookeeper, kafka and mongodb services.

One other thing to consider in our docker-compose file is the secrets. Each service may have a secret that need to be passed to other services. We use docker-compose secrets capability. When we define a secret for a service, Docker will create a file for that secret inside the container at /run/secrets/secret-name path. for example, We have defined a secret for auth-access-public-key. We pass this secret to other services that may need authentication service public key for verifying tokens signed by authentication service.

secrets:
  auth-access-public-key:
    file: auth/auth-service/keys/access_token.public.pem

Open a terminal, cd to play-microservices folder and run docker-compose up -d --build. Wait until all services run. Go to http://localhost:3000/ and voila! Our application is ready to be tested.

To do

deploy to a CI/CD environment via Jenkins
Add and run tests

Play Microservices: Client service

Khaled Hosseini — Tue, 18 Jul 2023 15:07:10 +0000

This is the 7th part of a series of articles under the name "Play Microservices". Links to other parts:
Part 1: Play Microservices: Bird's eye view
Part 2: Play Microservices: Authentication
Part 3: Play Microservices: Scheduler service
Part 4: Play Microservices: Email service
Part 5: Play Microservices: Report service
Part 6: Play Microservices: Api-gateway service
Part 7: You are here
Part 8: Play Microservices: Integration via docker-compose
Part 9: Play Microservices: Security

The source code for the project can be found here:

In the previous stages, we successfully developed all of back-end services including the auth, scheduler, email, report and api-gateway services. Our current objective is to establish a client service that acts as a single entry point for end users to access our application. As we independently develop the client service, the remaining services are unavailable to us during development. To overcome this limitation, we create mock implementations to simulate the behavior of the unavailable services during the development of the client service.

At the end, the project directory structure will appear as follows:

Tools

The tools required In the host machine:

Docker: Containerization tool
VSCode: Code editing tool
Dev containsers extension for VSCode
Docker extension for VSCode
Git

The tools and technologies that we will use Inside containers for each service:

Wiremock service: Wiremock. We use this service to simulate our api-gateway service.
Web-Client service:
- Typescript : programming language
- Next.js: Web framework that can be used for developing backend and front-end applications.
- Tailwind css: A utility-first CSS framework
- react-hook-form
- TanStack Query: A asynchronous state management.

Docker dev environment

Development inside Docker containers can provide several benefits such as consistent environments, isolated dependencies, and improved collaboration. By using Docker, development workflows can be containerized and shared with team members, allowing for consistent deployments across different machines and platforms. Developers can easily switch between different versions of dependencies and libraries without worrying about conflicts.

When developing inside a Docker container, you only need to install Docker, Visual Studio Code, and the Dev Containers and Dockerextensions on VS Code. Then you can run a container using Docker and map a host folder to a folder inside the container, then attach VSCode to the running container and start coding, and all changes will be reflected in the host folder. If you remove the images and containers, you can easily start again by recreating the container using the Dockerfile and copying the contents from the host folder to the container folder. However, it's important to note that in this case, any tools required inside the container will need to be downloaded again. Under the hood, When attaching VSCode to a running container, Visual Studio code install and run a special server inside the container which handle the sync of changes between the container and the host machine.

API mock service: Wiremock

During the development of our microservice application, we have implemented various patterns to ensure efficient development. One of the patterns we have followed is service-per-team development. This approach focuses on each team developing their services independently, with limited knowledge of and no direct access to other services. When our service relies on another service that is inaccessible during development, we use mocking techniques to simulate the behavior of that service. For different service communication protocols such as gRPC, REST API, GraphQL, and others, we have various applications and even online services that do this job for us. In our current scenario, we aim to replicate the behavior of an API gateway service. To achieve this, we rely on the usage of Wiremock, a tool that offers a convenient solution for mocking APIs. By utilizing Wiremock, we can simulate the responses and behavior of the API gateway service, allowing us to continue development and testing seamlessly. Using Wiremock and similar tools, we can effectively emulate the behavior of external services, enabling smoother development and testing workflows within our microservice architecture. Running and configuring Wiremock via Docker is quite easy. Lets begin!

Create a folder for the project and choose a name for it (such as 'microservice'). Then create a folder named client-web. This folder is the root directory of the current project. You can then open the root folder in VS Code by right-clicking on the folder and selecting 'Open with Code'.

Inside the root directory create a folder with the name wiremock, then create a Dockerfile and set content to FROM wiremock/wiremock:2.35.0

Create a folder inside wiremock named mappings. Inside this folder we define our endpoints. For example lets say we have the following endpoint in our api-gateway:

Then using a json file containing a request and a corresponding response we can configure wiremock to mimic that end-point.

{
    "request": {
      "method": "GET",
      "urlPattern": "/api/v1/ping"
    },
    "response": {
      "status": 200,
      "body": "{\"Message\": \"Pong\"}"
    }
}

-During the development of the client service, our API gateway service is not accessible. However, the protocol layer models for each service have already been determined and made accessible through collaborative efforts and guidance from the Technical Leads. We use this models to mock our api-gateway service. Copy all json files from here to mapping folder.

Create a file named .env in the root directory and add the following content:

WIREMOCK_PORT = 8088
WIREMOCK_CONTAINER_PORT=8080
CLIENT_PORT = 3000

Inside root directory create a file named docker-compose.yml and add the following content.

version: '3'
services:
  wiremock:
    build: 
      context: ./wiremock
      dockerfile: Dockerfile
    container_name: wriremock
    ports:
      - ${WIREMOCK_PORT}:${WIREMOCK_CONTAINER_PORT}
    volumes:
      - ./wiremock/mappings:/home/wiremock/mappings

Run docker-compose up -d --build. Now go to http://localhost:8088/__admin/. You can see the available mock end points.

you can test this service using applications like postman.

Our api-gateway mock is ready! run docker-compose down

Client-web service: Typescript

Before delving into the specifics, let's begin by describing the application we are developing. The application consists of four backend services: authentication, job scheduler, and report services. These services are not directly accessed by the client application. Instead, an API gateway acts as a bridge, facilitating communication between the client and these services.
The client application offers the following functionalities:

User Registration and Login: The application provides signup and login pages where users can create and authenticate their accounts.
Admin Features: Administrative users have additional capabilities, including:
- Querying the List of Registered Users: Admins can retrieve information about the registered users.
- Querying the List of Scheduled Jobs: Admins have the ability to inquire about the jobs currently scheduled and edit or delete them.
- Scheduling New Jobs: Admins can create and schedule new jobs.
- Querying Reports: Admins can retrieve reports generated by the system.

Our Next.js application structure can be summarized as follow:

A server that acts as a proxy between the api gateway and the client app. This server is optional and one benefit of it is Solving CORS problems as our api-gateway and web-client service may run on different domains. Also we can add an extra layer of security in this server to protect our-api gateway.

Three main pages that all are rendered in the client side (inside browser). Signup, login and main page. the main page has three components. One for querying the users list, One for handling jobs and the final one for querying reports.

Create a folder named client-service inside client-web folder.

Create a Dockerfile inside client-service and set the contents to

FROM node:20.4.0

WORKDIR /usr/src/app

Add the following to the service part of our docker-compose.yml file.

    client:
    build: 
      context: ./client-service
      dockerfile: Dockerfile
    container_name: client
    command: sleep infinity
    ports:
      - ${CLIENT_PORT}:${CLIENT_PORT}
    environment:
      - APIGATEWAY_URL=http://wriremock:${WIREMOCK_CONTAINER_PORT}
    volumes:
      - ./client-service:/usr/src/app

We are going to do all the development inside a docker container without installing node.js in our host machine. To do so, we run the containers and then attach VSCode to the client-service container. As you may noticed, the Dockerfile for client-service has no entry-point therefore we set the command value of it to sleep infinity to keep the container awake.

Now run docker-compose up -d --build

While running, attach to the client service by clicking bottom-left icon and then select attach to running container. Select client service and wait for a new instance of VSCode to start. Upon starting the attached instance of VSCode, you will be prompted to open a folder within the container. As we have designated the WORKDIR as /usr/src/app inside the Dockerfile, we will select this folder inside the container. It is important to note that this designated folder is mounted to the client-service folder on the host machine using Docker Compose volumes. Consequently, any changes made within the selected folder will be automatically synced to this folder on the host machine. This synchronization ensures that modifications made during development are reflected in both the container and the host environment.

After opening the folder /usr/src/app, open a new terminal and initialize the next.js project by running npx create-next-app --typescript client. You need to go through a list of question before initializing the project. Select the answers as shown below.

After initializing the app, a file named packages.json is created. This file contains information such as dependencies, package name, version, etc. Another part of this file is the scripts.

  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },

We can run this scripts from the client folder using npm run <scriptname>. Run cd client and then run npm run dev. Go to http://localhost:3000 and you can see the default page of our project. stop the service by hitting ctl + c

We use newly introduced app routing in our project. To compare app-routing and page routing refer to next.js website and read the documentation by selecting them.

We start by creating our server. This server will act as a proxy between our website and the api gateway. Create a folder named config inside src folder and then a file named index.tsx. set the content to

export const URL_APIGATEWAY=`${process.env.APIGATEWAY_URL}/api/v1`;

In the app router routing method, You can create a folder XXX inside the app folder and then create a file named rout.tsx and a file named page.tsx inside. The route.tsx acts as your api end point and the page.tsx is a webpage at that route. Create a folder named api and then a folder inside named ping. create a file named route.tsx inside ping and set the content to:

import { NextRequest, NextResponse } from "next/server";

export async function GET(req: NextRequest) {
    return new NextResponse(JSON.stringify({ message: "Pong" }), {
        status: 200,
      })
}

Run npm run dev. Now go http://localhost:3000/api/ping. you can see the response saying pong!

Stop the server by hitting ctl + c

Create a file named middleware.tsx inside src folder and set the content to

import { NextRequest, NextResponse } from 'next/server'
import {URL_APIGATEWAY} from './config'

export async function middleware(req: NextRequest) {
  const regex = new RegExp('/api/*')
  if (!regex.test(req.url)) {
    return new NextResponse(null, {
      status: 400,
      statusText: "Bad Request"
    })
  }
  console.log("middleware is called for url: ",req.url)
  const url = URL_APIGATEWAY + "/" + req.url.split("/api/")[1]
  console.log("middleware sends the request to : ", url)
  const res = await fetch( url, {
    method:req.method,
    headers: req.headers,
    body: req.body
  });

  return res
}

export const config = {
  matcher: '/api/:path*',
}

This middleware runs only for requests to the '/api/:path*' path. When we request this api, it will get the response from the api-gateway.

Run npm run dev. Now go http://localhost:3000/api/ping. you can see the response saying message: Pong.. This time the result has been returned from the api-gateway (Our mock service).

Stop the service by hitting ctl + c

Create a folder named types inside src. Inside this folder we will define our models. Copy all the files from here to this folder.

run npm install @tanstack/react-query react-hook-form react-hot-toast

Create a folder named components and then a folder named providers inside it. Here we are going to put the components that provide a capability for a context. Copy the files from here. One of the providers is AuthProvider. This component provides a context for the logged-in user to be used by any child elements inside the app.

'use client';

import {User} from '@/types'
import React from 'react';

const UserContext = React.createContext<
  [User | null, React.Dispatch<React.SetStateAction<User | null>>] | undefined
>(undefined);

export function AuthProvider({ children }: { children: React.ReactNode }) {
  const [user, setUser] = React.useState<User | null>(null);
  return (
    <UserContext.Provider value={[user, setUser]}>
      {children}
    </UserContext.Provider>
  );
}

export function useAuth() {
  const context = React.useContext(UserContext);
  if (context === undefined) {
    throw new Error('useAuth must be used within a UserContext');
  }
  return context;
}

We warp the whole pages in our application inside these providers. go to layout.tsx inside the app folder and change the file content to:

import './globals.css';
import type { Metadata } from 'next'
import { Inter } from 'next/font/google'
import { QueryProvider } from '@/components/providers/query_provider'
import {AuthProvider} from '@/components/providers/auth_provider'
import ToastProvider from '@/components/providers/toast_provider'

const inter = Inter({ subsets: ['latin'] })

export const metadata: Metadata = {
  title: 'Microservices job scheduler',
  description: 'A simple job scheduler app with microservices architecture.',
}

export default async function RootLayout({
  children,
}: {
  children: React.ReactNode
}) {

  return (
    <html lang="en">
      <body className={`${inter.className}  font-inter antialiased bg-gray-300 text-gray-900 tracking-tight`}>
      <ToastProvider />
        <AuthProvider>
          <QueryProvider>
            {children}
          </QueryProvider>
        </AuthProvider>
      </body>
    </html>
  )
}

Create a folder named client inside components folder and then a file named signup_form.tsx. Set the contents from here. Inside this file we use useForm for binding form content to CreateUserRequestmodel and useMutation to perform user registration.

Create a folder named (auth) (with parenthesis: this name will be ignored in the routing) and then a folder named signup. then create a file named page.tsx and set the content to

'use client'

import { useAuth } from "@/components/providers/auth_provider";
import SignUpForm from "@/components/client/signup_form";
import { redirect } from 'next/navigation'

export default function SignUp() {

  const [user,_setUser] = useAuth()
  if (user) {
      redirect("/")
  }

  return (
    <main>
      <SignUpForm/>
    </main>
  );
}

here we have used useAuth() to check if the user is already logged in or not. If yes, simply redirect to the homepage. Now run npm run dev and then go to http://localhost:3000/signup. Fill in the form to match the mapping of our mock api (In this case name: admin,email:admin@admin.com,password: password,role: 0). Now hit signup. If everything goes according to plan, the sign up would be successful and you will be redirected to the login page which does not exist at the moment.

Stop the server by hitting ctl + c

Create a file named login_form.tsx inside components/client folder and set the contents from here.

Create a folder inside (auth) folder named login and then a file named page.tsx. Set the contents from here.Now run npm run dev and then go to http://localhost:3000/login. Fill in the form to match the mapping of our mock api (in this case: email: admin@admin.com,password: password). If everything goes according to plan, the login would be successful and you will be redirected to the homepage.

Stop the server by hitting ctl + c

Create a folder named logout inside (auth) folder. Then a file named page.tsx. set the contents from here.
Create a folder named (default) inside the app folder and then move the page.tsx file from app folder to (default). This file would be our homepage. We first create neccessary components and then show them inside our home page.

Create a file named header.tsx inside components/client. Set the contents from here. Inside this component, we will show some items based on the logging state of the user. If the user is logged in, we show user's email and logout button. if the user is not logged in, we show login and signup buttons.

Create a folder named lib inside src folder. Then a file named api_gateway.tsx. We define a function called fetch_with_refresh_token. This function checks the return code for calls to our protected end-points and if the result is unauthorized, then we refresh the access token. Set the contents to:

import { StatusCodes } from 'http-status-codes';
// we do fetch, If the result is unauthorized, Possibily our access token has exired! we simply refresh it.
export async function fetch_with_refresh_token(url: string, options?: RequestInit | undefined): Promise<Response> {
    let result = await fetch(url,options)
    console.log("fetch result status code and texts are: ",result.status, result.statusText)
    if (result.status == StatusCodes.UNAUTHORIZED) {
        //possibly expired access token.
        console.log("Unauthorized. possibly access token has expired. lets refresh access token....")
        result = await fetch('/api/user/refresh_token', {
            method: "POST"
        })
        console.log("refresh token result arrived",result)
        if (result.status == StatusCodes.OK) {
            console.log("refresh token request is successfull. calling again.")
            //we have accessed new access token in browser cookie. now request again with our new access token
            result = await fetch(url,options)
            return result
        }else {
            console.log("refresh token request was not successfull")
        }
    }
    return result
}

Change the contents of page.tsx inside (default) folder to

'use client'

import React, { useEffect, useState } from "react"
import { ParseUser } from "@/types";
import { useAuth } from "@/components/providers/auth_provider";
import Header from '@/components/client/header'
import {fetch_with_refresh_token} from '../../lib/api_gateway'

export default function Home() {

  const [_user,setUser] = useAuth()

  useEffect(() => {
    // declare the data fetching function
    const fetchData = async () => {
      const userResponse = await fetch_with_refresh_token("/api/user/get");
      const user = await ParseUser(userResponse)
      console.log("Home.useEffect.fetchData: User is: ",user)
      setUser(user)
    }

    fetchData()
    .catch(console.error);
  }, [setUser])

  const [currentComponent, setCurrentComponent] = useState<string | null>(null);

  return (
    <div>
      <Header/>
      <div className="flex min-h-screen flex-row bg-gray-100 text-gray-800">
        <main className="main -ml-48 flex flex-grow flex-col p-4 transition-all duration-150 ease-in md:ml-0">
          Main
        </main>
      </div>
    </div>
  );
}

Now run npm run dev. Then go to http://localhost:3000/ and you can see the header. You can now navigate through the pages. Go to http://localhost:3000/signup. Enter credentials and you will be redirected to login page. Then fill in the form and after hitting login button you will be redirected to the home page. As you are logged in now, the email and the logout button will be shown on the header.

Now it is time to create other components of our home page. Copy the remaining files from here to components/client folder. This files are components for user, job and report handlings. Set the content of page.tsx inside (default) folder from here.

Run npm run dev. Go to http://localhost:3000/ and voila. Our web app is ready.

To DO

Add tests

Add tracing using Jaeger

Add monitoring and analysis using grafana

Refactoring

I would love to hear your thoughts. Please comment on your opinions. If you found this helpful, let's stay connected on Twitter! xaledhosseini.

Play Microservices: Api-gateway service

Khaled Hosseini — Tue, 11 Jul 2023 12:41:02 +0000

This is the 6th part of a series of articles under the name "Play Microservices". Links to other parts:
Part 1: Play Microservices: Bird's eye view
Part 2: Play Microservices: Authentication
Part 3: Play Microservices: Scheduler service
Part 4: Play Microservices: Email service
Part 5: Play Microservices: Report service
Part 6: You are here
Part 7: Play Microservices: Client service
Part 8: Play Microservices: Integration via docker-compose
Part 9: Play Microservices: Security

The source code for the project can be found here:

In the previous stages, we successfully developed a set of services consisting of the auth, scheduler, email, and report services. Our current objective is to establish a gateway service that acts as a single entry point for clients to access these individual services. To ensure smooth development and testing, we have included an additional service in our development environment solely for debugging purposes. As we independently develop the API gateway, the remaining services are temporarily unavailable. To overcome this limitation, we create mock implementations to simulate the behavior of the unavailable services during the development of the API gateway.

At the end, the project directory structure will appear as follows:

Tools

The tools required In the host machine:

Docker: Containerization tool
VSCode: Code editing tool
Dev containsers extension for VSCode
Docker extension for VSCode
Git

The tools and technologies that we will use Inside containers for each service:

Gripmock service: Gripmock
Api-gateway service:
- Golang : programming language
- gRPC-GO: gRPC framework for golang
- Gin: Is a web framework written in Golang
- gin-swagger for our rest api documentation.

Docker dev environment

API mock service: Gripmock

We have 3 services in our microservice application that the api-gateway service will communicate with them. Authentication service, Job scheduler service and report service. These services are all gRPC services. During the development of api-gateway we do not have access to these services, therefore we mock their behavior. For this purpose we use gripmock. Instructions on how these service works can be found here. For gripmock configuration, we need .proto files for our gRPC services. Then we add stubs to the stub service. For this we can either define our stubs inside a json file (static stubbing) or add stubbing on the fly with a simple REST API (HTTP stub server is running on port :4771). Lets begin.

Create a folder for the project and choose a name for it (such as 'microservice'). Then create a folder named api-gateway. This folder is the root directory of the current project. You can then open the root folder in VS Code by right-clicking on the folder and selecting 'Open with Code'.

Inside the root directory create a folder with the name gripmock, then create the following files inside.

Create a Dockerfile and set content to FROM tkpd/gripmock:v1.12.1

Copy all .proto files from here to the gripmock directory.

Create a file named .env in the root directory and add the following content:

GRIPMOCK_ADMIN_PORT=4771
GRIPMOCK_GRPC_PORT=4770
API_GATEWAY_PORT=5010

Inside gripmock folder create a folder named stubs. Inside these folder we are going to create json stubs for our services. you can copy the files from here. For example, for report.json we have the following content. Our grpc service name is ReportService and the method name is ListReports. For input, we define some equals critria, that is when we received a request for service ReportService and method ListReports and the values for filter, page and size are 1, 1, and 10 respectively, then return the following output. For more information see here.

[
  {
    "service": "ReportService",
    "method": "ListReports",
    "input": {
      "equals": {
        "filter": 1,
        "page": 1,
        "size": 10
      }
    },
    "output": {
      "data": "..."
    }
  }
]

Inside root directory create a file named docker-compose.yml and add the following content.

version: '3'
services:
  gripmock:
    build:
      context: ./gripmock
      dockerfile: Dockerfile
    container_name: gripmock
    ports:
      - ${GRIPMOCK_GRPC_PORT}:${GRIPMOCK_GRPC_PORT}
      - ${GRIPMOCK_ADMIN_PORT}:${GRIPMOCK_ADMIN_PORT}
    volumes:
      - ./gripmock:/mock
    # we use admin to manage grpc servers. we connect to grpc servers using our clinets(from code).
    command: > 
      --admin-listen=0.0.0.0 
      --admin-port=${GRIPMOCK_ADMIN_PORT} 
      --grpc-listen=0.0.0.0 
      --grpc-port=${GRIPMOCK_GRPC_PORT} 
      --stub=/mock/stub /mock/job.proto /mock/report.proto /mock/user.proto

Run docker-compose up -d --build. Now go to http://localhost:4771/. You can see the available mock grpc servers. These services acts as a real grpc servers. We connect them from our code and query them.

Run docker-compose down

Api-gateway service: Golang

We aim to develop an API gateway service that acts as a REST API server and a gRPC client. This service will receive REST requests from client applications and connect to various gRPC servers to fetch the required results. The API gateway acts as an intermediary that facilitates communication between the client and the gRPC servers. A summary of what we are going to do: First, we will import the .proto files for the gRPC servers and compile them into the Go programming language using protoc. This step ensures that we have the necessary client libraries to interact with the gRPC services.
Next, we'll prepare gRPC clients for each individual gRPC service. These clients will allow our API gateway to establish connections and communicate with the respective gRPC servers. Finally, we will run our REST API service, which will expose multiple endpoints for client applications to interact with. Within each endpoint, we will utilize a specific gRPC client to handle the requests and retrieve data from the corresponding gRPC server.

Create a folder named api-gateway-service inside api-gateway folder.

Create a Dockerfile inside api-gateway-service and set the contents to

FROM golang:1.19

ENV PROTOC_VERSION=23.3
ENV PROTOC_ZIP=protoc-${PROTOC_VERSION}-linux-x86_64.zip
RUN apt-get update && apt-get install -y unzip
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/$PROTOC_ZIP \
    && unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
    && unzip -o $PROTOC_ZIP -d /usr/local 'include/*' \ 
    && rm -f $PROTOC_ZIP
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
RUN go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
RUN export PATH="$PATH:$(go env GOPATH)/bin"

WORKDIR /usr/src/app

Add the following to the service part of our docker-compose.yml file.

  api-gateway:
    build: 
      context: ./api-gateway-service
      dockerfile: Dockerfile
    container_name: api-gateway
    environment:
      - SERVER_PORT=${API_GATEWAY_PORT}
      - CLIENT_DOMAIN=localhost # we call the server from this domain (for cookie registration ,...)
      - AUTH_SERVICE_URL=gripmock:${GRIPMOCK_GRPC_PORT}
      - SCHEDULER_SERVICE_URL=gripmock:${GRIPMOCK_GRPC_PORT}
      - REPORT_SERVICE_URL=gripmock:${GRIPMOCK_GRPC_PORT}
      - AUTH_PUBLIC_KEY_FILE=/run/secrets/auth-public-key
    ports:
      - ${API_GATEWAY_PORT}:${API_GATEWAY_PORT}
    command: sleep infinity
    volumes:
      - ./api-gateway-service:/usr/src/app

We are going to do all the development inside a docker container without installing Golang in our host machine. To do so, we run the containers and then attach VSCode to the api-gateway-service container. As you may noticed, the Dockerfile for api-gateway-service has no entry-point therefore we set the command value of it to sleep infinity to keep the container awake.

Now run docker-compose up -d --build

While running, attach to the api-gateway service by clicking bottom-left icon and then select attach to running container. Select api-gateway service and wait for a new instance of VSCode to start. Upon starting the attached instance of VSCode, you will be prompted to open a folder within the container. As per our Dockerfile configuration, we have designated the WORKDIR as /usr/src/app. Therefore, we will select this folder inside the container. It is important to note that this designated folder is mounted to the api-gateway-service folder on the host machine using Docker Compose volumes. Consequently, any changes made within the selected folder will be automatically synced to the corresponding folder on the host machine. This synchronization ensures that modifications made during development are reflected in both the container and the host environment.

After opening the folder /usr/src/app, open a new terminal and initialize the go project by running go mod init github.com/<your_username>/play-microservices/api-gateway/api-gateway-service. This command will create a go.mod file.

Run go get -u google.golang.org/grpc. This is a gRPC framework for running grpc server using Golang.

Now create a folder named proto and copy the proto files from here.

Create file named build_grpc.sh and set the contents to:

#!/bin/bash
declare -a services=("proto")
for SERVICE in "${services[@]}"; do
   DESTDIR='proto'
   mkdir -p $DESTDIR
   protoc \
       --proto_path=$SERVICE/ \
       --go_out=$DESTDIR \
       --go-grpc_out=$DESTDIR \
       $SERVICE/*.proto
done

Run source build_grpc.sh. This command compile our .proto file to Golang. For each .proto file, Two files will be created. X.pb.go and X_grpc.pb.go. The first contains the proto models and the second contains the code for grpc service interface.

Note: We adopt a Golang project structure that aligns with the recommended guidelines stated here

Create a folder named config and a file named config.go. Set the contents from here. Also create a file named .env in the same folder. we will put our internal environment variables here. Set contents from here.

Create a folder named pkg in the root directory (beside mod.go). We will put general packages here. Inside Create a folder named logger then a file named logger.go and set the contents from here.

Inside pkg folder create a folder named grpc and then a file named GRPC_Client.go. set the content to:

package grpc

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

type GRPC_Client struct{}

func (jc *GRPC_Client) Connect(url string) (*grpc.ClientConn, error) {
    return grpc.Dial(url, grpc.WithTransportCredentials(insecure.NewCredentials()))
}

Create a folder named cookie inside pkg, then a file named cookie.go. Set the contents from here.

Create this folder tree: internal/models/report/grpc. inside grpc folder create a file named client_service.go and set the contents to

package grpc

import (
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/config"
    gr "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/pkg/grpc"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/pkg/logger"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/proto"
    "github.com/gin-gonic/gin"
)

type ReportGRPCClient struct {
    log logger.Logger
    cfg *config.Config
    gr.GRPC_Client
}

func NewReportGRPCClient(log logger.Logger, cfg *config.Config) *ReportGRPCClient {
    return &ReportGRPCClient{log: log, cfg: cfg}
}

func (jc *ReportGRPCClient) GRPC_ListReports(c *gin.Context, listReportsRequest *proto.ListReportsRequest) (*proto.ListReportResponse, error) {
    jc.log.Info("ReportGRPCClient.GRPC_ListReports: Connecting to grpc server...")
    conn, err := jc.Connect(jc.cfg.ReportServiceURL)
    if err != nil {
        return nil, err
    }
    defer conn.Close()

    client := proto.NewReportServiceClient(conn)
    jc.log.Info("ReportGRPCClient.GRPC_ListReports: Conneced to grpc server...")
    jc.log.Infof("ReportGRPCClient.GRPC_ListReports: calling server for ListReports: %v", listReportsRequest)
    return client.ListReports(c, listReportsRequest)
}

This file contains the logic for creating a client for report gRPC service and then retrieve the list of reports. We will use this class in our rest handler.

Run go install github.com/swaggo/swag/cmd/swag@latest We will use this tool for generation of rest api documentations.

Run go get -u github.com/swaggo/gin-swagger and go get -u github.com/swaggo/files

Create a folder named handler inside report folder and then a file named handler.go. set the contents to

package handler

import (
    "net/http"
    "strconv"

    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/config"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/internal/models"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/internal/models/report/grpc"
    grpcutils "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/pkg/grpc"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/pkg/logger"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/proto"
    "github.com/gin-gonic/gin"
)

type ReportHandler struct {
    log logger.Logger
    *grpc.ReportGRPCClient
}

func NewReportHandler(log logger.Logger, cfg *config.Config) *ReportHandler {
    grpcClient := grpc.NewReportGRPCClient(log, cfg) // Initialize the embedded type
    return &ReportHandler{log: log, ReportGRPCClient: grpcClient}
}

// @Summary Get the list of reports
// @Description retrieve the reports
// @Tags report
// @Produce json
// @Param   page      query    int     true        "Page"
// @Param   size      query    int     true        "Size"
// @Success 200 {array} models.ListReportResponse
// @Router /report/list [get]
func (rh *ReportHandler) ListReports(c *gin.Context) {
    rh.log.Infof("Request arrived: list reports: %v", c.Request)

    page, err := strconv.ParseInt(c.Query("page"), 10, 32)
    if err != nil {
        rh.log.Error("ReportHandler.ListReports: invalid input. page is not provided in the query")
        c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid page parameter"})
        return
    }

    size, err := strconv.ParseInt(c.Query("size"), 10, 32)
    if err != nil {
        rh.log.Error("ReportHandler.ListReports: invalid input. size is not provided in the query")
        c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid size parameter"})
        return
    }

    res, err := rh.GRPC_ListReports(c.Request.Context(), &proto.ListReportsRequest{Page: page, Size: size})
    if err != nil {
        rh.log.Errorf("ReportHandler.ListReports: error listing reports: %v",err.Error())
        status := grpcutils.GetHttpStatusCodeFromGrpc(err)
        c.AbortWithStatusJSON(status, gin.H{"status": false, "error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, models.ListReportResponseFromProto(res))
}

Run swag init -g ./cmd/main.go -o ./docs. This command will parse comments and generate required files(docs folder and docs/doc.go) at /docs folder.

Create a folder named api inside internal folder and then a file named router.go. set the contents to

package api

import (
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/config"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/docs"
    rh "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/internal/models/report/handler"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/pkg/logger"
    "github.com/gin-gonic/gin"

    "net/http"

    swaggerfiles "github.com/swaggo/files"
    ginSwagger "github.com/swaggo/gin-swagger"
)

type Router struct {
    log logger.Logger
    cfg *config.Config
}

func NewRouter(log logger.Logger, cfg *config.Config) *Router {
    return &Router{log: log, cfg: cfg}
}
func (r *Router) Setup(router *gin.Engine) {

    docs.SwaggerInfo.BasePath = "/api/v1"
    router.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerfiles.Handler))

    router.GET("/api/v1/ping", r.Pong)

    reportHandler := rh.NewReportHandler(r.log, r.cfg)
    router.GET("/api/v1/report/list", reportHandler.ListReports)
}

// @BasePath /api/v1
// PingExample godoc
// @Summary ping example
// @Schemes
// @Description do ping
// @Tags ping
// @Accept json
// @Produce json
// @Success 200 {string} Pong
// @Router /ping [get]
func (s *Router) Pong(c *gin.Context) {
    c.JSON(http.StatusOK, "Pong")
}

Create a folder named server inside internal folder. Then a file named server.go. set the content to

package server

import (
    "fmt"

    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/config"
    api "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/internal/api"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/pkg/logger"

    "github.com/gin-gonic/gin"
)

type Server struct {
    log logger.Logger
    cfg *config.Config
}

func NewServer(log logger.Logger, cfg *config.Config) *Server {
    return &Server{log: log, cfg: cfg}
}

func (s *Server) Run() {
    r := gin.Default()

    router := api.NewRouter(s.log, s.cfg)
    router.Setup(r)

    r.Run(fmt.Sprintf("0.0.0.0:%s", s.cfg.ServerPort))
}

Create a folder named cmd and a file named main.go. Set the content to

package main

import (
    "log"

    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/config"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/internal/server"
    "github.com/<yourusername>/play-microservices/api-gateway/api-gateway-service/pkg/logger"
)

func main() {

    cfg, err := config.InitConfig()
    if err != nil {
        log.Fatal(err)
    }

    appLogger := logger.NewApiLogger(cfg)
    appLogger.InitLogger()
    appLogger.Infof(
        "AppVersion: %s, LogLevel: %s, Environment: %s",
        cfg.AppVersion,
        cfg.Logger_Level,
        cfg.Environment,
    )
    appLogger.Infof("Success parsed config: %#v", cfg.AppVersion)

    appLogger.Info("Starting the server")
    s := server.NewServer(appLogger, cfg)
    s.Run()
}

Run go mod tidy

Run go run cmd/main.go

Open a browser and go to http://localhost:5010/api/v1/report/list?page=1&size=10 If everything goes on plan, you can connect to the server and receive the results. What happens here is that from the browser we send a rest request to our api-gateway by calling http://localhost:5010/api/v1/report/list?page=1&size=10. The gateway receive our parameters, and then create a gRPC request to our mock server. The mock server returns the result and then the api-gateway return that result to us.

Stop the server by hitting ctl + c

We repeat the exact procedure for user and job services. For user and job you can get the files from here and here respectively. Also do not forget to add the endpoints to router.go file from here.

It is essential to consider the clear separation of REST API models and gRPC models. Just as we adhere to the practice of separating the database layer models from the gRPC layer models in our gRPC services, we follow the same approach here. By separating the models specific to the REST layer from those used in the gRPC layer, we gain valuable flexibility in querying and combining data from multiple gRPC services. This distinction allows us to define and manage models unique to each layer in a cohesive manner, facilitating efficient data retrieval and manipulation within the respective contexts.

Some notes on swagger and rest api parameter types. We use gin-swagger for creating documentation for our rest api. In rest api we have 5 parameter types:

Query parameters: appended to the URL after a question mark (?). example: http://localhost:5010/api/v1/report/list?page=1&size=10

Path parameters: are part of the URL path. Example: http://localhost:5010/api/v1/user/{id}

Headers: are key-value pairs included in the request or response headers. Example: Authorization, Content-Type.

The request body: carries additional data sent with the HTTP request. It is used to send complex or larger data payloads, such as JSON or XML, to the server.

Form data: is used to submit data from an HTML form to the server. It consists of key-value pairs representing form fields and their values. Form data is encoded and sent in the body of an HTTP request with the "Content-Type" header set as "application/x-www-form-urlencoded" or "multipart/form-data".

For example the swag declarative comments for GetUser endpoint is as follows. For more details see here

// @Summary Get user
// @Description Get user
// @Tags user
// @Produce json
// @Success 200 {object} models.GetUserResponse
// @Router /user/get [get]
func (uh *UserHandler) GetUser(c *gin.Context) {

    res, err := uh.GRPC_GetUser(c, &proto.GetUserRequest{})

    if err != nil {
        c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"status": false, "error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, models.GetUserResponseFromProto(res))
}

After adding swag comments Run swag init --parseDependency -g ./cmd/main.go -o ./docs and then run go run cmd/main.go. Go to http://localhost:5010/swagger/index.html and you can see the swagger documentation for your rest api. based on the stubs you have created for your gRPC servers, you can test the api endpoints.

In the end, we will add an auth interceptor to our api-gateway. Create a folder named interceptors inside internal/api and then a file named auth_interceptor.go.

package interceptors

import (
    "net/http"

    "github.com/KhaledHosseini/play-microservices/api-gateway/api-gateway-service/pkg/cookie"
    "github.com/gin-gonic/gin"
    "google.golang.org/grpc/metadata"
)

// We do authentication and authorization in the end services. We just attach the auth headers to grpc requests.
func AuthenticateUser() gin.HandlerFunc {
    return func(c *gin.Context) {
        accessToken, err := cookie.GetAccessToken(c) // Retrieve the access token from the request header
        if err != nil {
            c.AbortWithStatusJSON(http.StatusForbidden, gin.H{"status": false, "error": err.Error()})
            return
        }

        md := metadata.Pairs("authorization", accessToken)
        ctx := metadata.NewOutgoingContext(c.Request.Context(), md)

        // Update the request context with the modified context
        c.Request = c.Request.WithContext(ctx)

        c.Next()
    }
}

In microservices architecture, depending on the design one approach is to do authentication inside api-gateway and authorization inside downstream services. If we use another layer of authentication between microservices using protocols like mTLS, this approach can be considered safe. Another approach is to do both authentication and authorization of users inside downstream services. Here we use the second approach and inside interceptor file we just add the authorization header of http call to the metadata header of grpc context. In gRPC, headers are sent as part of the gRPC message metadata using key-value pairs. The headers are serialized using Protocol Buffers (protobuf) and encoded as binary data while REST API headers are sent as part of the HTTP request or response headers. They are typically represented as plain text in a key-value format, following the HTTP header specifications.

Change the contents of router.go inside api folder to add the interceptor.

func (r *Router) Setup(router *gin.Engine) {

    docs.SwaggerInfo.BasePath = "/api/v1"
    router.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerfiles.Handler))

    router.GET("/api/v1/ping", r.Pong)

    userHandler := uh.NewUserHandler(r.log, r.cfg)
    jobHandler := jh.NewJobHandler(r.log, r.cfg)
    reportHandler := rh.NewReportHandler(r.log, r.cfg)

    router.POST("/api/v1/user/create", userHandler.CreateUser)
    router.POST("/api/v1/user/login", userHandler.LoginUser)
    router.POST("/api/v1/user/refresh_token", userHandler.RefreshAccessToken)
    router.POST("/api/v1/user/logout", userHandler.LogOutUser)

    // Apply the middleware to the routes inside the router.Group function
    api := router.Group("/api")
    api.Use(interceptors.AuthenticateUser()) // Apply the middleware here
    {
        api.GET("/v1/user/get", userHandler.GetUser)
        api.GET("/v1/user/list", userHandler.ListUsers)

        api.POST("/v1/job/create", jobHandler.CreateJob)
        api.POST("/v1/job/update", jobHandler.UpdateJob)
        api.GET("/v1/job/get", jobHandler.GetJob)
        api.GET("/v1/job/list", jobHandler.ListJobs)
        api.POST("/v1/job/delete", jobHandler.DeleteJob)

        api.GET("/v1/report/list", reportHandler.ListReports)
    }
}

For subsequent requests, you need to add authorization headers to the request. In this case to be able to access protected endpoints, you need to login first. On logging in the cookie for the user will be set on the browser. You can inspect the cookies value by right click on the page and select inspect. Then under application tab and storage section you can see the values for authorizationand x-refresh-token values. On subsequent requests, these cookies will be sent along with the request. Api-gateway then add the authorization cookie to the metadata of grpc context to be consumed by downstream services for authentication and authorization purposes.

To DO

Add tests

Add tracing using Jaeger

Add monitoring and analysis using grafana

Refactoring

I would love to hear your thoughts. Please comment on your opinions. If you found this helpful, let's stay connected on Twitter! xaledhosseini.

Play Microservices: Report service

Khaled Hosseini — Tue, 04 Jul 2023 23:39:01 +0000

This is the 5th part of a series of articles under the name "Play Microservices". Links to other parts:
Part 1: Play Microservices: Bird's eye view
Part 2: Play Microservices: Authentication
Part 3: Play Microservices: Scheduler service
Part 4: Play Microservices: Email service
Part 5: You are here
Part 6: Play Microservices: Api-gateway service
Part 7: Play Microservices: Client service
Part 8: Play Microservices: Integration via docker-compose
Part 9: Play Microservices: Security

The source code for the project can be found here:

Summary
Tools
Docker dev environment
Developement services
- Database service: Mongodb
- Mongo express service
- Kafka metadata service: Zookeeper
- Zoonavigator service
- Message broker service: Kafka
- Kafka-ui service
- Report-grpcui-service
Report service: Python
To do

Summary

Up to now we have developed 4 services. Now, our objective is to create a report service for our microservices application. This service is responsible for saving all events in our application. It listens to all kafka events and save them in a database. To develop this service , we need four distinct services: a database service, a message broker service, a metadata database service dedicated to supporting the message broker, and the report service itself, which is a gRPC API service. Additionally, in the development environment, we include four extra services specifically for debugging purposes. These services consist of Mongo Express, used to manage our database service, Kafkaui for managing our Kafka service, Zoonavigator for the Zookeeper service, and grpcui for testing our gRPC API.

At the end, the project directory structure will appear as follows:

Tools

The tools required In the host machine:

Docker: Containerization tool
VSCode: Code editing tool
Dev containsers extension for VSCode
Docker extension for VSCode
Git

The tools and technologies that we will use Inside containers for each service:

Database service: Mongo
Mongo express service: Mongo express
Messaging service: Kafka
Kafka-ui service: Kafka-ui
Metadata service: Zookeeper
Zoonavigator service: Zoonavigator
grpcui service: grpcui
report service:
- Python : programming language
- grpc: gRPC framework for python
- Pymongo: Query builder for our database communication.
- Kafka-Python for our message broker communications from python.
- grpcio-tools For compiling .proto files to python.

Docker dev environment

Development services

Create a folder for the project and choose a name for it (such as 'microservice'). Then create a folder named report. This folder is the root directory of current project. You can then open the root folder in VS Code by right-clicking on the folder and selecting 'Open with Code'.

Our approach is to develop all services of our microservice application independently, and as development services are Identical to the part 3 you can copy the folders for development services from here. Do not forget to also copy docker-compose.yml file.

Report service: Python

Our goal is to create a python service that consumes kafka events for all topics in our application (produced by other services) and then store them in a database. The admin then can query the reports. To respond to the queries, we run an grpc service and for listening to kafka events we subscribe to kafka events using kafka consumers. lets begin.

Create a folder named report-service inside report folder.

Create a Dockerfile inside report-service and set the contents to

FROM python:3.11.4

RUN apt-get update \
  && apt-get install -y --no-install-recommends graphviz libsnappy-dev \
  && rm -rf /var/lib/apt/lists/* \
  && pip install --no-cache-dir pyparsing pydot

WORKDIR /usr/src/app

Inside report-service folder create a folder named keys and then go to this link, generate a key pair with length 2048. Create a file named access_token.public.pem inside the keys folder and then paste the public key. Keep the private key somewhere else. We will use it later. This public key acts as the public key of authentication service and we are going to verify the JWTs with it.

Add the following to the service part of our docker-compose.yml file.(if you have copied the file, It already has this data).

  report-service:
    build: 
      context: ./report-service
      dockerfile: Dockerfile
    container_name: report-service
    command: sleep infinity
    ports:
      - ${REPORT_SERVICE_PORT}:${REPORT_SERVICE_PORT}
    environment:
      ENVIRONMENT: development
      SERVER_PORT: ${REPORT_SERVICE_PORT}
      DATABASE_USER_FILE: /run/secrets/report-db-user
      DATABASE_PASS_FILE: /run/secrets/report-db-pass
      DATABASE_DB_NAME_FILE: /run/secrets/report-db-dbname
      DATABASE_SCHEMA: mongodb
      DATABASE_HOST_NAME: report-db-service
      DATABASE_PORT: ${MONGODB_PORT}
      KAFKA_BROKERS: r-kafka1-service:${KAFKA1_PORT}
      AUTH_PUBLIC_KEY_FILE: /run/secrets/auth-public-key
      # TOPICS_FILE: ''
    volumes:
      - ./report-service:/usr/src/app
    depends_on:
      - r-kafka1
    secrets:
      - report-db-user
      - report-db-pass
      - report-db-dbname
      - auth-public-key

We are going to do all the development inside a docker container without installing Python in our host machine. To do so, we run the containers and then attach VSCode to the report-service container. As you may noticed, the Dockerfile for report-service has no entry-point therefore we set the command value of report-service to sleep infinity to keep the container awake.

Now run docker-compose up -d --build

While running, attach to the report-service by clicking bottom-left icon and then select attach to running container. Select report-service and wait for a new instance of VSCode to start. At the beginning the VScode asks us to open a folder inside the container. We have selected WORKDIR /usr/src/app inside our Dockerfile, so we will open this folder inside the container. This folder is mounted to report-service folder inside the host machine using docker compose volume (see volumes of report-service inside docker-compose), therefor whatever change we made will be synced to the host folder too.

After opening the folder /usr/src/app, create a file named requirements.txt and set the contents to:

kafka-python==2.0.2
python-snappy==0.6.1
crc32c==2.3.post0
python-dotenv==1.0.0
pymongo==4.4.0
grpcio-tools==1.56.0
grpcio-reflection==1.56.0
python_jwt==4.0.0

Open a new terminal and run pip install -r requirements.txt

Create a file named .env.topics and set the contents to:

TOPIC_JOB_RUN="topic-job-run"
TOPIC_JOB_RUN_CONSUMER_GROUP_ID="topic-job-run-consumer-report"
TOPIC_JOB_RUN_CONSUMER_WORKER_COUNT=1
TOPIC_JOB_RUN_RESULT="topic-job-run-result"
TOPIC_JOB_RUN_RESULT_CONSUMER_GROUP_ID="job-run-result-consumer-report"
TOPIC_JOB_RUN_RESULT_CONSUMER_WORKER_COUNT=1
TOPIC_JOB_CREATE="topic-job-create"
TOPIC_JOB_CREATE_CONSUMER_GROUP_ID="topic-job-create-consumer-report"
TOPIC_JOB_CREATE_CONSUMER_WORKER_COUNT= 1
TOPIC_JOB_UPDATE="topic-job-update"
TOPIC_JOB_UPDATE_CONSUMER_GROUP_ID="topic-job-update-consumer-report"
TOPIC_JOB_UPDATE_CONSUMER_WORKER_COUNT=1

Create a file named config.py and set the contents from here.

create a folder named app then a file named __init__.py for the app module. set the contents to

from config import Config

def create_app():
    config = Config()

Create a file named main.py in the root directory and set the contents to

from app import create_app
import logging

def main():
    create_app()

if __name__ == '__main__':
    logging.basicConfig(
        format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:' + '%(levelname)s:%(process)d:%(message)s',
        level=logging.INFO
    )

    main()

A summary of what we are going to do: First we define a .proto file for our report. Then we compile the .proto file to python. As it is common for separating protocol models from database models, we define database models and implement the logic for converting to/from protocol models. Then we create a report service and implement the grpc protocol interfaces. This report service read the reports from the database and send it to the end user. Next part is to prepare a database for our report service for reading and writing the reports and finally we prepare our kafka consumer classes to listen to kafka events and then save them to reports database. The folder structure for our service is as follows:

Create a folder named proto and set the contents from here. Now create a file named build_grpc.sh and set the content to:

#!/bin/bash

declare -a services=("proto")
for SERVICE in "${services[@]}"; do
    DESTDIR='proto'
    mkdir -p $DESTDIR
    python -m grpc_tools.protoc \
        --proto_path=$SERVICE/ \
        --python_out=$DESTDIR \
        --grpc_python_out=$DESTDIR \
        $SERVICE/*.proto
done

Now run source build_grpc.sh. This will compile .proto files and store them in proto directory(As this is a linux command, be careful about the line endings to be LF and not CRLF). Add a file named __init__.py file and set the contents to the following. We do this to avoid repeating ugly file names when importing them in other modules.

import proto.report_pb2_grpc as ReportGRPC
import proto.report_pb2 as ReportGRPCTypes

Also change the import report_pb2 as report__pb2 inside generated report_pb2_grpc.py to import proto.report_pb2 as report__pb2 to avoid import errors later.

Create a folder named models inside app folder. We have two models here. One is Report which we transfer via gRPC and the other is Job which we receive via message events. Here we develop our logic on top of our models. For the Report model, we need a database and a grpc service. We define the Report model inside model module and the database and grpc services inside another module called report. For the job model we need a consumer service to read it from kafka events and as the same logic for report we define the job model itself inside models module and the consumer service inside another module called job.

Create two files named _report.py and _job.py and set the contents from here and here respectively. We Add _ to the files to indicate them as internal files. We expose them via __init__.py file. Then set the contents of the __init__.py file inside models folder to

import json

class JsonObject(object):
    def toJsonData(self):
        return self.toJsonStr().encode('utf-8')

    def toJsonStr(self):
        return json.dumps(self.to_dict())

    def to_dict(self):
        return vars(self)

from app.models._job import Job
from app.models._report import Report
from app.models._report import ReportList
from app.models._report import ReportDBInterface

Create a folder named reportand then two folders named db and grpc. As we are going to use Mongo for our database, first create another folder inside app called database_engines and then a file named _mongo.py. set the content from here. Then expose it from __init__.py file using from app.database_engines._mongo import MongoDB Inside db folder create a folder named _report_db_mongo.py and set the content from here. This file contains ReportDBMongo which has inherited from two classes. One is MongoDB and the other is ReportDBInterface.

class ReportDBMongo(MongoDB, ReportDBInterface):
    def __init__(self, cfg: Config):
        super().__init__(cfg)
        ReportDBInterface.__init__(self)
        self.dbName = cfg.DatabaseDBName
        self.collection = "ReportsCollection"
        self.db = self.client[self.dbName]

Inside grpc folder, create a file named _report_grpc_service.py. This file contains our grpc service and implements our grpc protocol interface. Set the contents to:

from proto import ReportGRPC
import logging

from app.models import ReportDBInterface

class MyReportService(ReportGRPC.ReportService):
    def __init__(self, reportDB: ReportDBInterface):
        self.reportDB = reportDB
        super().__init__()

    def ListReports(self,request, context):
        logging.info(f"message recieved...{request}")
        filter = request.filter
        page = request.page
        size = request.size
        result = self.reportDB.list(type=filter,page=page,size=size)
        return result.toProto()

Now it is time to run our job consumer service. As we are going to use kafka-python for our messaging service, first create a folder named message_engines and then a file named _kafka_consumer_worker.py. set the contents from here. This class contains the logic to create and run a kafka consumer for a specific topic. It receives topic name, consumer group id and the list of kafka broker addresses and then subscribes to the topic. whenever a new message arrives messageRecieved function will be called. We inherit this class and override messageRecieved function and do our logic.

class KafkaConsumerWorker:
    def __init__(self, topic: str, consumerGroupId: str, kafkaBrokers: List[str]):
        self.Topic = topic
        self.ConsumerGroupId = consumerGroupId
        self.KafkaBrokers = kafkaBrokers
    def run_kafka_consumer(self):
        consumer = KafkaConsumer(self.Topic,
        group_id=self.ConsumerGroupId, 
        bootstrap_servers=self.KafkaBrokers,
        enable_auto_commit=False,
        value_deserializer= self.loadJson)

        logging.info("consumer is listening....")
        try:
            for message in consumer:
                json = self.loadJson(message.value)
                self.messageRecieved(self.Topic, json)
                #committing message manually after reading from the topic
                consumer.commit()
        except Exception as e:
            logging.error(f"consumer listener stoped with error: {e}")
        finally:
            consumer.close()

Inside models/job folder create another folder named message_broker and then a file named _job_consumer_service.py. Inside this file we run consumers for all topics that their messages are a job model. Set the content from here. One important class in this file is JobConsumerWorkerthat has inherited from KafkaConsumerWorker. We have overridden the messageRecieved function and when a new job arrives, we create a new report in the ReportDB for it.

class JobConsumerWorker(KafkaConsumerWorker):
    def __init__(
        self,
        topic: str,
        consumerGroupId: str,
        kafkaBrokers: List[str],
        reportDB: ReportDBInterface,
    ):
        super().__init__(
            topic=topic, consumerGroupId=consumerGroupId, kafkaBrokers=kafkaBrokers
        )
        self.reportDB = reportDB

    def messageRecieved(self, topic: str, message: any):
        try:
            job = Job.from_dict(message)
            report = Report(
                type=0,
                topic=topic,
                created_time=datetime.now(),
                report_data=job.toJsonStr(),
            )
            self.reportDB.create(report=report)
        except Exception as e:
            logging.error("Unable to load json for job ", e)

Expose ConsumerService inside __init__.py file using from app.models.job.message_broker._job_consumer_service import ConsumerService

Our services are ready. Now lets run them. Create a folder named server inside app and the a file named __init__.py. Set the contents to

from concurrent import futures
import logging

import grpc
from grpc_reflection.v1alpha import reflection

from proto import ReportGRPC
from proto import ReportGRPCTypes

from config import Config

from app.models.report import MyReportService
from app.models.report import ReportDBMongo
from app.models.job import ConsumerService

class Server:
    def run(self, cfg: Config):
        logging.info("Running services...")
        db = ReportDBMongo(cfg)
        my_report_grpc_service = MyReportService(reportDB=db)
        server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
        ReportGRPC.add_ReportServiceServicer_to_server(my_report_grpc_service, server)
        SERVICE_NAMES = (
            ReportGRPCTypes.DESCRIPTOR.services_by_name["ReportService"].full_name,
            reflection.SERVICE_NAME,
        )
        reflection.enable_server_reflection(SERVICE_NAMES, server)
        port = cfg.ServerPort
        server.add_insecure_port("[::]:" + port)
        server.start()

        consumer = ConsumerService(cfg,db)
        consumer.start()


        server.wait_for_termination()

Now go to __init__.py of app folder and change the contents to

from config import Config
from app.server import Server

import logging
def create_app():
    logging.info("creating app...")
    config = Config()
    s = Server()
    s.run(config)

Finally it is time to run and test our server! run python main.py. If everything goes according to plan, both our grpc server and our kafka consumers are running. Go to http://localhost:8080/ and under the topics, create a new topic called topic-job-run (we have defined this named in environment variables). Then click on the topic and produce a message for it by clicking on the top right corner button and pasting the following json content to the value of the message and then clicking produce message.

{
    "Id": "649f07e619fca8aa63d842f6",
    "Name": "job1",
    "ScheduleTime": "2024-01-01T00:00:00Z",
    "CreatedAt": "2023-06-30T16:50:46.3042083Z",
    "UpdatedAt": "2023-06-30T16:50:46.3042086Z",
    "Status": 2,
    "JobData": {
        "SourceAddress": "example@example.com",
        "DestinationAddress": "example@example.com",
        "Subject": "subject ",
        "Message": "message"
    }
}

This message will be consumed by our app and a new report item will be created for it. To confirm, go to http://localhost:8081/. in the Mongo Express panel click reports_db and then view. Our report item has been created!

You can repeat and reproduce some more messages the same way.

Now lets test our grpc server and read our reports. First go to docker desktop and be sure that our grpcui service is running and attached to the service. (restart the service if necessary). Go to http://localhost:5000/. Fill in the form by selecting job for filter, 0 for page and 10 for page size and click invoke! if everything goes according to plan you will receive the results!

The last step is to add authentication and authorization to our report service. We first check the authenticity of the JWT to see if it is issued by our auth service or not. then we read the role of inside JWT and if it is admin, we allow access to the reports.

Create a folder named grpc_interceptorsinside app folder. then a file named _token_validation_interceptor.py inside grpc_interceptors folder. Set the contents of file from here. This file contains an iterceptor that our grpc calls go through before reaching grpc server methods. We read the authorization header (which is a JWt). then we verify it and check for admin role.

def intercept_service(self, continuation, handler_call_details):
        # Exclude reflection service from authentication so that it can retrieve the service info
        if handler_call_details.method == "/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo":  
            return continuation(handler_call_details)

        # Extract the token from the gRPC request headers
        metadata = dict(handler_call_details.invocation_metadata)
        token = metadata.get('authorization', '')
        if token:
            logging.info(f"Token found: {token}")
            public_key = jwk.JWK.from_pem(self.cfg.AuthPublicKey)
            header, claims = jwt.verify_jwt(token, public_key, ['RS256'], checks_optional=True,ignore_not_implemented=True)
            roleStr = claims['role'].lower()
            if roleStr == "admin":
                logging.info(" role is Admin: Authorized")
                return continuation(handler_call_details)
            return self._notAuthorized
        else:
            logging.info("No token found")
            return self._noAuthHeader

Create a file named __init__.py inside grpc_interceptors and set the contents to

from app.grpc_interceptors._token_validation_interceptor import TokenValidationInterceptor

Change the server definition inside server init file to match the following:

        from app.grpc_interceptors import TokenValidationInterceptor
        ...

        server = grpc.server(
            futures.ThreadPoolExecutor(max_workers=10),
            interceptors = (TokenValidationInterceptor(cfg),)
        )

Run python main.py. Now an auth interceptor in active and protect our grpc server. If you invoke any method, the response would be Missing Authorization header

go to this site and put your generated rsa key-pairs in the corresponding boxes. Then define a simple jwt like this:

{
  "iss": "microservice-scheduler.com",
  "sub": "sheniqua",
  "aud": "maxine",
  "iat": 1689904636,
  "exp": 1689910603,
  "role": "admin"
}

Set other configs as shown in the following image and then click the left arrow. Copy the encoded token to the clipboard.

Now go to docker desktop and restart grpcui-service then go to http://localhost:5000/. For each request you make, set a metadata with name= authorization and the value= <token_copied>

This time when you invoke the methods, authentication will pass. you can change the role to user and regenerate the token and use the new token. This time the response would be Not authorized happy debugging :)

To DO

Add tests

Add tracing using Jaeger

Add monitoring and analysis using grafana

Refactoring

I would love to hear your thoughts. Please comment on your opinions. If you found this helpful, let's stay connected on Twitter! xaledhosseini.

Play Microservices: Email service

Khaled Hosseini — Sun, 02 Jul 2023 10:49:18 +0000

This is the 4th part of a series of articles under the name "Play Microservices". Links to other parts:
Part 1: Play Microservices: Bird's eye view
Part 2: Play Microservices: Authentication
Part 3: Play Microservices: Scheduler
Part 4: You are here
Part 5: Play Microservices: Report service
Part 6: Play Microservices: Api-gateway service
Part 7: Play Microservices: Client service
Part 8: Play Microservices: Integration via docker-compose
Part 9: Play Microservices: Security

The source code for the project can be found here:

In the 3rd part, we developed a Scheduler service. Now, our objective is to create an Email job executer service that consumes events of topic-job-run produced by scheduler service and after running the job, produce topic-job-run-result to the message broker. To achieve this, we need four distinct services: a message broker service, a metadata database service dedicated to supporting the message broker, an smtp-server service and the email service itself. Additionally, in the development environment, we include two extra services specifically for debugging purposes. These services consist of Kafkaui for managing our Kafka service, Zoonavigator for the Zookeeper service. Please keep in mind that we are following service per team pattern, assuming each service in our microservice architecture has its own repository and development team and in the development environment they are completely independent (Technically not logically). The first four parts are a copy from the previous step. You can skip them and got to SMTP service

In the end, the project directory structure will appear as follows:

Tools

The tools required In the host machine:

Docker: Containerization tool
VSCode: Code editing tool
Dev containsers extension for VSCode
Docker extension for VSCode
Git

The tools and technologies that we will use Inside containers for each service:

Messaging service: Kafka
Kafka-ui service: Kafka-ui
Metadata service: Zookeeper
Zoonavigator service: Zoonavigator
SMTP service: Postfix
Email service:
- Python : programming language
- Kafka-python: For our message broker communications from go.

Docker dev environment

Metadata service: Zookeeper

ZooKeeper is a centralized service for maintaining configuration information. we use it as metadata storage for our Kafka messaging service.

Inside root directory create a folder with the name zookeeper-service

Create a Dockerfile and set content to FROM bitnami/zookeeper:3.8.1

Create a file named .env and set content to

ZOO_SERVER_USERS=admin,user1
# for development environment only
ALLOW_ANONYMOUS_LOGIN="yes"
# if yes, uses SASL
ZOO_ENABLE_AUTH="no"

Create a file named server_passwords.properties and set content to password123,password_for_user1 Please choose your own passwords.

Add the following to the .env file of the docker-compose (the .env file at the root directory of the project.)

ZOOKEEPER_PORT=2181
ZOOKEEPER_ADMIN_CONTAINER_PORT=8078
ZOOKEEPER_ADMIN_PORT=8078

Add the following to the service part of the docker-compose.yml.

  e-zk1:
    build:
      context: ./zookeeper-service
      dockerfile: Dockerfile
    container_name: e-zk1-service
    secrets:
      - zoo-server-pass
    env_file:
      - ./zookeeper-service/.env
    environment:
      ZOO_SERVER_ID: 1
      ZOO_SERVERS: e-zk1-service:${ZOOKEEPER_PORT}:${ZOOKEEPER_PORT}
      ZOO_SERVER_PASSWORDS_FILE: /run/secrets/zoo-server-pass
      ZOO_ENABLE_ADMIN_SERVER: yes
      ZOO_ADMIN_SERVER_PORT_NUMBER: ${ZOOKEEPER_ADMIN_CONTAINER_PORT}
    ports:
      - '${ZOOKEEPER_PORT}:${ZOOKEEPER_PORT}'
      - '${ZOOKEEPER_ADMIN_PORT}:${ZOOKEEPER_ADMIN_CONTAINER_PORT}'
    volumes:
      - "e-zookeeper_data:/bitnami"

volumes:
  e-zookeeper_data:
    driver: local

Add the following to the secrets part of the docker-compose.yml.

  zoo-server-pass:
    file: zookeeper-service/server_passwords.properties

-ZooKeeper is a distributed application that allows us to run multiple servers simultaneously. It enables multiple clients to connect to these servers, facilitating communication between them. ZooKeeper servers collaborate to handle data and respond to requests in a coordinated manner. In this case, our zookeeper consumers (clients) are Kafka servers which is again a distributed event streaming platform. We can run multiple zookeeper services as an ensemble of zookeeper servers and attach them together via ZOO_SERVERS environment variable.

The Bitnami ZooKeeper Docker image provides a zoo_client entrypoint, which acts as an internal client and allows us to run the zkCli.sh command-line tool to interact with the ZooKeeper server as a client. But we are going to use a GUI client for debugging purposes: Zoonavigator.

Zoonavigator service

This service exists only in the development environment for debugging purposes. We use it to connect to zookeeper-service and manage the data.

Inside root directory create a folder with the name zoonavigator-service

Create a Dockerfile and set content to FROM elkozmon/zoonavigator:1.1.2

Add ZOO_NAVIGATOR_PORT=9000 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  e-zoo-navigator:
    build: 
      context: ./zoonavigator-service
      dockerfile: Dockerfile
    container_name: e-zoo-navigator-service
    ports:
      - '${ZOO_NAVIGATOR_PORT}:${ZOO_NAVIGATOR_PORT}'
    environment:
      - CONNECTION_LOCALZK_NAME = Local-zookeeper
      - CONNECTION_LOCALZK_CONN = localhost:${ZOOKEEPER_PORT}
      - AUTO_CONNECT_CONNECTION_ID = LOCALZK
    depends_on:
      - e-zk1

Now from the terminal run docker-compose up -d --build

While running go to http://localhost:9000/. You will see the following screen:

Enter the container name of a zookeeper service (here e-zk1). If everything goes according to plan, you should be able to establish a connection to the ZooKeeper service.

No run docker-compose down. We will return to these tools later.

Message broker service: Kafka

Apache Kafka is an open-source distributed event streaming platform that is well-suited for Microservices architecture. It is an ideal choice for implementing patterns such as event sourcing. Here We use it as an message broker for our scheduler service.

Inside root directory create a folder with the name kafka-service

Create a Dockerfile and set content to FROM bitnami/kafka:3.4.1

Create a .env file beside the Docker file and set the content to:

ALLOW_PLAINTEXT_LISTENER=yes
KAFKA_ENABLE_KRAFT=no

Add KAFKA1_PORT=9092 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  e-kafka1:
    build: 
      context: ./kafka-service
      dockerfile: Dockerfile
    container_name: e-kafka1-service
    ports:
      - '${KAFKA1_PORT}:${KAFKA1_PORT}'
    volumes:
      - "e-kafka_data:/bitnami"
    env_file:
      - ./kafka-service/.env
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_CFG_ZOOKEEPER_CONNECT: zk1:${ZOOKEEPER_PORT}
      KAFKA_ZOOKEEPER_PROTOCOL: PLAINTEXT #if auth is enabled in zookeeper use one of: SASL, SASL_SSL see https://hub.docker.com/r/bitnami/kafka
      KAFKA_CFG_LISTENERS: PLAINTEXT://:${KAFKA1_PORT}
    depends_on:
      - e-zk1

To connect to our Kafka brokers for debugging purposes, we run another service. Kafka-ui.

Kafka-ui service

This service exists only in the development environment for debugging purposes. We use it to connect to kafka-service and manage the data.

Inside root directory create a folder with the name kafkaui-service

Create a Dockerfile and set content to FROM provectuslabs/kafka-ui:latest

Add KAFKAUI_PORT=8080 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  e-kafka-ui:
    build: 
      context: ./kafkaui-service
      dockerfile: Dockerfile
    container_name: e-kafka-ui-service
    restart: always
    ports:
      - ${KAFKAUI_PORT}:${KAFKAUI_PORT}
    environment:
     KAFKA_CLUSTERS_0_NAME: local
     KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: e-kafka1:${KAFKA1_PORT}
     DYNAMIC_CONFIG_ENABLED: 'true'
    depends_on:
      - e-kafka1

Now run docker-compose run -d --build. While containers are running, go to http://localhost:8080/ to open Kafka-ui dashboard.

From the interface, you have the ability to view and manage brokers, topics, and consumers. We'll revisit these elements in more detail shortly.

Run docker-compose down

SMTP service: OPTIONAL

We want to run a Simple Outbound Email Service on our local computer via docker. This step is optional and it is preferred to use 3rd party services like Amazon SES.

Inside root directory create a folder with the name postfix-service

Create a Dockerfile and set content to FROM FROM catatnight/postfix:latest

Add POSTFIX_PORT=25 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

    postfix:
    build: 
      context: ./postfix-service
      dockerfile: Dockerfile
    container_name: postfix
    restart: always
    environment:
      - EMAIL_DOMAIN=yourdomain.com
      - SMTP_USER=username
      - SMTP_PASSWORD=password
    ports:
      - ${POSTFIX_PORT}:${POSTFIX_PORT}

Our required services are ready and running. Now it is time to Prepare development environment for our email job executer service.

Mailjob service: Python

Our goal is to create a simple python service that consumes kafka events for topic topic-job-run (produce by our scheduler service), execute the job (sending an email) and then produce an event for topic topic-job-run-result. A piece of cake!

Create a folder named mailjob-service inside scheduler folder.

Create a Dockerfile inside scheduler-service and set the contents to

FROM python:3.11.4

WORKDIR /usr/src/app

Add the following to the service part of our docker-compose.yml file.

  mailjob-service:
    build: 
      context: ./mailjob-service
      dockerfile: Dockerfile
    container_name: mailjob-service
    command: sleep infinity
    ports:
      - ${EMAIL_SERVICE_PORT}:${EMAIL_SERVICE_PORT}
    environment:
      ENVIRONMENT: development
      KAFKA_BROKERS: e-kafka1-service:${KAFKA1_PORT}
      # TOPICS_FILE: ''
      MAIL_SERVER_HOST: postfix
      MAIL_SERVER_PORT: 25
      EMAIL_DOMAIN: yourdomain.com
      SMTP_USER: username
      SMTP_PASSWORD: password

    volumes:
      - ./mailjob-service:/usr/src/app
    depends_on:
      - e-kafka1
      - postfix

We are going to do all the development inside a docker container without installing Python in our host machine. To do so, we run the containers and then attach VSCode to the mailjob-service container. As you may noticed, the Dockerfile for mailjob-service has no entry-point therefore we set the command value of mailjob-service to sleep infinity to keep the container awake.

Now run docker-compose up -d --build

While running, attach to the mailjob service by clicking bottom-left icon and then select attach to running container. Select mailjob-service and wait for a new instance of VSCode to start. At the beginning the VScode asks us to open a folder inside the container. We have selected WORKDIR /usr/src/app inside our Dockerfile, so we will open this folder inside the container. This folder is mounted to mailjob-service folder inside the host machine using docker compose volume, therefor whatever change we made will be synced to the host folder too.

After opening the folder /usr/src/app, create a file named requirements.txt and set the contents to:

kafka-python==2.0.2
python-dotenv==1.0.0

Open a new terminal and run pip install -r requirements.txt

Create a file named .env.topics and set the contents to:

TOPIC_JOB_RUN="topic-job-run"
TOPIC_JOB_RUN_CONSUMER_GROUP_ID="topic-job-run-consumer-emailexecutor"
TOPIC_JOB_RUN_CONSUMER_WORKER_COUNT=1
TOPIC_JOB_RUN_RESULT="topic-job-run-result"
TOPIC_JOB_RUN_RESULT_CREATE_PARTITIONS=1
TOPIC_JOB_RUN_RESULT_CREATE_REPLICAS=1

Create a file named config.py and set the contents from here.

create a folder named app then a file named __init__.py for the app module. set the contents to

from config import Config

def create_app():
    config = Config()

Create a file named main.py in the root directory and set the contents to

from app import create_app
import logging

def main():
    create_app()

if __name__ == '__main__':
    logging.basicConfig(
        format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:' + '%(levelname)s:%(process)d:%(message)s',
        level=logging.INFO
    )

    main()

Create a folder named email inside app, then a file named __init__.py and set the contents from here. This class is responsible for executing the email job.

Our goal is to consume kafka messages with the following structure.

{
    "Id": "649f07e619fca8aa63d842f6",
    "Name": "job1",
    "ScheduleTime": "2024-01-01T00:00:00Z",
    "CreatedAt": "2023-06-30T16:50:46.3042083Z",
    "UpdatedAt": "2023-06-30T16:50:46.3042086Z",
    "Status": 2,
    "JobData": {
        "SourceAddress": "example@example.com",
        "DestinationAddress": "example@example.com",
        "Subject": "subject ",
        "Message": "message"
    }
}

Create a folder named models inside app, then create __init__.py for the module. Set the contents to:

import json

class JsonObject:
    def toJsonData(self):
        return self.toJsonStr().encode('utf-8')

    def toJsonStr(self):
        return json.dumps(self.to_dict())

    def to_dict(self):
        return vars(self)

from app.models._email_job import EmailJob
from app.models._email_job import Email
from app.models._email_job import JobStatusEnum

Create a file named _email_job.py and set the contents from here.

Create this folder tree email_job/message_broker inside models folder. Then Create a file named __init__.py and set the contents from here. This file contains our kafka server and worker classes which consumes TOPIC_JOB_RUN, then runs the job and finally produce the TOPIC_JOB_RUN_RESULT topic. MessageBrokerService class accepts config and email executer in the constructor then Creates TOPIC_JOB_RUN_RESULT if it doesnot exists and finally runs kafka consumer workers.

    def __init__(self,cfg: Config, emailExecuter: EmailSender) -> None:
        self.cfg = cfg
        self.emailExecuter = emailExecuter
        # producers are responsible for creation of topics they produce
        admin_client = KafkaAdminClient(bootstrap_servers=cfg.KafkaBrokers)
        topic_exists = cfg.TopicJobRunResult in admin_client.list_topics()
        if not topic_exists:
            try:
                topic_list = []
                topic_list.append(NewTopic(name=cfg.TopicJobRunResult, num_partitions=cfg.TopicJobRunResultCreatePartitions, replication_factor=cfg.TopicJobRunResultCreateReplicas))
                admin_client.create_topics(new_topics=topic_list, validate_only=False)
                logging.info(f"Topic '{cfg.TopicJobRunResult}' created successfully.")
            except TopicAlreadyExistsError as e:
                logging.info(f"Topic '{cfg.TopicJobRunResult}' already exists. Error: {e}")
        else:
            logging.info(f"Topic '{cfg.TopicJobRunResult}' already exists.")


    def run(self):
        logging.info(f"Starting email job consumers with  {self.cfg.TopicJobRunWorkerCount } workers...")
        if self.cfg.TopicJobRunWorkerCount == 1:
            self.run_worker()
            logging.info(f"Worker 1 started for consuming job events...")
        else:
            worker_threads = []
            for i in range(0,self.cfg.TopicJobRunWorkerCount):
                t = Thread(target=self.worker)
                t.Daemon = True
                worker_threads.append(t)
                t.start()
                logging.info(f"Worker {i} started for consuming job events...")

            for t in worker_threads:
                t.join()

    def run_worker(self):
        job_consumer = JobConsumerWorker(self.cfg, self.emailExecuter)
        job_consumer.run()

The logic for running kafka consumer for TOPIC_JOB_RUN and producing TOPIC_JOB_RUN_RESULT is inside JobConsumerWorker class. We receive config and email sender class in the constructor and then run the consumer via following function:

    def run_kafka_consumer(self):
        consumer = KafkaConsumer(self.cfg.TopicJobRun,
        group_id=self.cfg.TopicJobRunConsumerGroupID, 
        bootstrap_servers=self.cfg.KafkaBrokers,
        value_deserializer= self.loadJson)

        for msg in consumer:
            if isinstance(msg.value, EmailJob):
                logging.info(f"An email job json recieved. handling the job. {msg.value}")
                self.handleJob(msg.value)
            else:
                logging.error(f"error handling: {msg}")

Create a folder inside app folder named server and a file named __init__.py. set the content to:

from config import Config
from app.email import EmailSender
from app.models.email_job.message_broker import MessageBrokerService

class Server:
    def run(self, cfg: Config):
        es = EmailSender(cfg)
        ecs = MessageBrokerService(cfg,es)
        ecs.run()

Now return to __init__.py of app folder and change the code to

import logging
from config import Config
from app.server import Server

def create_app():
    logging.info("Creating job-executor app")
    config = Config()
    s = Server()
    s.run(config)

Now run python run main.py If everything goes according to plan, our app starts and waiting for messages from kafka.

go to http://localhost:8080/. Under topics select topic-job-run. From top right, click on Produce message button and set the value to the following json and click Produce message.

{
    "Id": "649f07e619fca8aa63d842f6",
    "Name": "job1",
    "ScheduleTime": "2024-01-01T00:00:00Z",
    "CreatedAt": "2023-06-30T16:50:46.3042083Z",
    "UpdatedAt": "2023-06-30T16:50:46.3042086Z",
    "Status": 2,
    "JobData": {
        "SourceAddress": "example@example.com",
        "DestinationAddress": "example@example.com",
        "Subject": "subject ",
        "Message": "message"
    }
}

The app consumes the message, Then try to send the email and then produce another message for the topic topic-job-run-result that in our microservice app will be consumed by the scheduler-service. You can go to http://localhost:8080/ and under topics you can see that another topic has been created and it has one message. (Note: We have activated automatic topic creation in kafka in development environment. In production environment, it is common to have a separate service for topic creation and management for the whole microservice application).

To DO

Add tests

Add tracing using Jaeger

Add monitoring and analysis using grafana

Refactoring

I would love to hear your thoughts. Please comment on your opinions. If you found this helpful, let's stay connected on Twitter! xaledhosseini.

Play Microservices: Scheduler service

Khaled Hosseini — Fri, 30 Jun 2023 21:29:22 +0000

This is the 3rd part of a series of articles under the name "Play Microservices". Links to other parts:
Part 1: Play Microservices: Bird's eye view
Part 2: Play Microservices: Authentication
Part 3: You are here
Part 4: Play Microservices: Email service
Part 5: Play Microservices: Report service
Part 6: Play Microservices: Api-gateway service
Part 7: Play Microservices: Client service
Part 8: Play Microservices: Integration via docker-compose
Part 9: Play Microservices: Security

The source code for the project can be found here:

In the second part, we developed an authentication service. Now, our objective is to create a job scheduler service for our microservices application. To achieve this, we need four distinct services: a database service, a message broker service, a metadata database service dedicated to supporting the message broker, and the scheduler service itself, which is a gRPC API service. Additionally, in the development environment, we include four extra services specifically for debugging purposes. These services consist of Mongo Express, used to manage our database service, Kafkaui for managing our Kafka service, Zoonavigator for the Zookeeper service, and grpcui for testing our gRPC API.

At the end, the project directory structure will appear as follows:

Tools

The tools required In the host machine:

Docker: Containerization tool
VSCode: Code editing tool
Dev containsers extension for VSCode
Docker extension for VSCode
Git

The tools and technologies that we will use Inside containers for each service:

Database service: Mongo
Mongo express service: Mongo express
Messaging service: Kafka
Kafka-ui service: Kafka-ui
Metadata service: Zookeeper
Zoonavigator service: Zoonavigator
grpcui service: grpcui
Scheduler api service:
- Golang : programming language
- gRPC-GO: gRPC framework for golang
- Mongo driver: Query builder for our database communication.
- Kafka go for our message broker communications from go.
- Quartz for scheduling purposes.

Docker dev environment

Database service: Mongo

Create a folder for the project and choose a name for it (such as 'microservice'). Then create a folder named scheduler. This folder is the root directory of current project. You can then open the root folder in VS Code by right-clicking on the folder and selecting 'Open with Code'.

Inside the root directory create a folder with the name scheduler-db-service, then create the following files inside.

Create a Dockerfile and set content to FROM mongo:7.0.0-rc5

Create a file named pass.txt and set content to a password

Create a file named user.txt and set content to admin

Create a file named db_name.txt and set content to jobs_db

Create a file named .env in the root directory and set the content to MONGODB_PORT=27017.

Inside root directory create a file named docker-compose.yml and add the following content.

version: '3'
services:
# database service for scheduler service
  scheduler-db-service:
    build: 
      context: ./scheduler-db-service
      dockerfile: Dockerfile
    container_name: scheduler-db-service
    environment:
      MONGO_INITDB_ROOT_USERNAME_FILE: /run/secrets/scheduler-db-user
      MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/scheduler-db-pass
    env_file:
      - ./scheduler-db-service/.env
    ports:
      - ${MONGODB_PORT}:${MONGODB_PORT}
    secrets:
      - scheduler-db-user
      - scheduler-db-pass
      - scheduler-db-dbname 
    volumes:
      -  scheduler-db-service-VL:/data/db

volumes:
  scheduler-db-service-VL:

secrets:
  scheduler-db-user:
    file: scheduler-db-service/user.txt
  scheduler-db-pass:
    file: scheduler-db-service/pass.txt
  scheduler-db-dbname:
    file: scheduler-db-service/db_name.txt

In our Docker Compose file, we use secrets to securely share credential data between containers. While we could use an .env file and environment variables, this is not considered safe. When defining secrets in the Compose file, Docker creates a file inside each container (which has the secrets name) under the /run/secrets/ path, which the containers can then read and use. For example, we will set the path of the Docker Compose secret scheduler-db-pass to the DATABASE_PASS_FILE environment variable of scheduler service. The service then will go to the path (/run/secrets/scheduler-db-pass) and read the password file. We will be using these secrets in other services later in the project.

Mongo express service

The purpose of this service is solely for debugging and management of our running database server in the development environment.

Inside root directory create a folder with the name mongo-express-service

Create a Dockerfile and set content to FROM mongo-express:1.0.0-alpha.4

Create a file named .env beside Dockerfile and set the content to

ME_CONFIG_BASICAUTH_USERNAME=admin
ME_CONFIG_BASICAUTH_PASSWORD=password123
ME_CONFIG_MONGODB_ENABLE_ADMIN=true

Add the following lines to the .env file of the docker-compose (the .env file at the root directory of the project.)

MONGO_EXPRESS_PORT=8081

Add the following to the service part of the docker-compose.yml.

    mongo-express:
    build:
      context: ./mongo-express-service
      dockerfile: Dockerfile
    container_name: mongo-express-service
    restart: always
    environment:
      - ME_CONFIG_MONGODB_PORT=${MONGODB_PORT}
      - ME_CONFIG_MONGODB_SERVER=scheduler-db-service
      - ME_CONFIG_MONGODB_ADMINUSERNAME=root
      - ME_CONFIG_MONGODB_ADMINPASSWORD=password123
    env_file:
      - ./mongo-express-service/.env
    ports:
      - ${MONGO_EXPRESS_PORT}:${MONGO_EXPRESS_PORT}
    depends_on:
      - scheduler-db-service

Here, as mongo express doesn't provide a capability to read database password from the files, we simply pass mongodb credentials using environment variables (Do not forget we are in development environment).

Now open a terminal in your project directory and run docker-compose up. Docker Compose will download and cache the required images before starting your containers. For the first run, this may take a couple of minutes. If everything goes according to plan, you can then access the mongo express panel at http://localhost:8081/ and log in using the mongo express credentials from the .env file inside the mongo-express-service container. You should see that it has successfully connected to the scheduler-db-service container.

Now run docker-compose down

Metadata service: Zookeeper

ZooKeeper is a centralized service for maintaining configuration information. we use it as metadata storage for our Kafka messaging service.

Inside root directory create a folder with the name zookeeper-service

Create a Dockerfile and set content to FROM bitnami/zookeeper:3.8.1

Create a file named .env and set content to

ZOO_SERVER_USERS=admin,user1
# for development environment only
ALLOW_ANONYMOUS_LOGIN="yes"
# if yes, uses SASL
ZOO_ENABLE_AUTH="no"

Create a file named server_passwords.properties and set content to password123,password_for_user1 Please choose your own passwords.

Add the following to the .env file of the docker-compose (the .env file at the root directory of the project.)

ZOOKEEPER_PORT=2181
ZOOKEEPER_ADMIN_CONTAINER_PORT=8078
ZOOKEEPER_ADMIN_PORT=8078

Add the following to the service part of the docker-compose.yml.

  zk1:
    build:
      context: ./zookeeper-service
      dockerfile: Dockerfile
    container_name: zk1-service
    secrets:
      - zoo-server-pass
    env_file:
      - ./zookeeper-service/.env
    environment:
      ZOO_SERVER_ID: 1
      ZOO_SERVERS: zk1:${ZOOKEEPER_PORT}:${ZOOKEEPER_PORT} #,zk2:{ZOOKEEPER_PORT}:${ZOOKEEPER_PORT}
      ZOO_SERVER_PASSWORDS_FILE: /run/secrets/zoo-server-pass
      ZOO_ENABLE_ADMIN_SERVER: yes
      ZOO_ADMIN_SERVER_PORT_NUMBER: ${ZOOKEEPER_ADMIN_CONTAINER_PORT}
    ports:
      - '${ZOOKEEPER_PORT}:${ZOOKEEPER_PORT}'
      - '${ZOOKEEPER_ADMIN_PORT}:${ZOOKEEPER_ADMIN_CONTAINER_PORT}'
    volumes:
      - "zookeeper_data:/bitnami"

Add the following to the secrets part of the docker-compose.yml.

  zoo-server-pass:
    file: zookeeper-service/server_passwords.properties

-ZooKeeper is a distributed application that allows us to run multiple servers simultaneously. It enables multiple clients to connect to these servers, facilitating communication between them. ZooKeeper servers collaborate to handle data and respond to requests in a coordinated manner. In this case, our zookeeper consumers (clients) are Kafka servers which is again a distributed event streaming platform. We can run multiple zookeeper services as an ensemble of zookeeper servers and attach them together via ZOO_SERVERS environment variable.

The Bitnami ZooKeeper Docker image provides a zoo_client entrypoint, which acts as an internal client and allows us to run the zkCli.sh command-line tool to interact with the ZooKeeper server as a client. But we are going to use a GUI client for debugging purposes: Zoonavigator.

Zoonavigator service

This service exists only in the development environment for debugging purposes. We use it to connect to zookeeper-service and manage the data.

Inside root directory create a folder with the name zoonavigator-service

Create a Dockerfile and set content to FROM elkozmon/zoonavigator:1.1.2

Add ZOO_NAVIGATOR_PORT=9000 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  zoo-navigator:
    build: 
      context: ./zoonavigator-service
      dockerfile: Dockerfile
    container_name: zoo-navigator-service
    ports:
      - '${ZOO_NAVIGATOR_PORT}:${ZOO_NAVIGATOR_PORT}'
    environment:
      - CONNECTION_LOCALZK_NAME = Local-zookeeper
      - CONNECTION_LOCALZK_CONN = localhost:${ZOOKEEPER_PORT}
      - AUTO_CONNECT_CONNECTION_ID = LOCALZK
    depends_on:
      - zk1

Now from the terminal run docker-compose up -d --build

While running go to http://localhost:9000/. You will see the following screen:

Enter the container name of a zookeeper service (here zk1). If everything goes according to plan, you should be able to establish a connection to the ZooKeeper service.

No run docker-compose down. We will return to these tools later.

Message broker service: Kafka

Inside root directory create a folder with the name kafka-service

Create a Dockerfile and set content to FROM bitnami/kafka:3.4.1

Create a .env file beside the Docker file and set the content to:

ALLOW_PLAINTEXT_LISTENER=yes
KAFKA_ENABLE_KRAFT=no

Add KAFKA1_PORT=9092 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  kafka1:
    build: 
      context: ./kafka-service
      dockerfile: Dockerfile
    container_name: kafka1-service
    ports:
      - '${KAFKA1_PORT}:${KAFKA1_PORT}'
    volumes:
      - "kafka_data:/bitnami"
    env_file:
      - ./kafka-service/.env
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:${KAFKA1_PORT},LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:${KAFKA1_PORT}
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_CFG_ZOOKEEPER_CONNECT: zk1:${ZOOKEEPER_PORT}
      KAFKA_ZOOKEEPER_PROTOCOL: PLAINTEXT #if auth is enabled in zookeeper use one of: SASL, SASL_SSL see https://hub.docker.com/r/bitnami/kafka
      KAFKA_CFG_LISTENERS: PLAINTEXT://:${KAFKA1_PORT}
    depends_on:
      - zk1

In order to connect to our Kafka brokers for debugging purposes, we run another service. Kafka-ui.

Kafka-ui service

This service exists only in the development environment for debugging purposes. We use it to connect to kafka-service and manage the data.

Inside root directory create a folder with the name kafkaui-service

Create a Dockerfile and set content to FROM provectuslabs/kafka-ui:latest

Add KAFKAUI_PORT=8080 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  kafka-ui:
    build: 
      context: ./kafkaui-service
      dockerfile: Dockerfile
    container_name: kafka-ui-service
    restart: always
    ports:
      - ${KAFKAUI_PORT}:${KAFKAUI_PORT}
    environment:
     KAFKA_CLUSTERS_0_NAME: local
     KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka1:${KAFKA1_PORT}
     DYNAMIC_CONFIG_ENABLED: 'true'
    depends_on:
      - kafka1

Now run docker-compose run -d --build. While containers are running, go to http://localhost:8080/ to open Kafka-ui dashboard.

From the interface, you have the ability to view and manage brokers, topics, and consumers. We'll revisit these elements in more detail shortly.

Run docker-compose down

Our required services are ready and running. Now it is time to Prepare development environment for our scheduler service.

Scheduler-grpcui-service

Before commencing the development of our Scheduler service, let's incorporate an additional service into our development environment. This service will facilitate interaction with our scheduler-service for debugging purposes.

Create a folder named grpcui-service inside scheduler folder.

Create a Docker file and set contents to FROM fullstorydev/grpcui:v1.3.1

Add the following to the services part of the docker-compose.yml file.

  grpcui-service:
    build:
      context: ./grpcui-service
      dockerfile: Dockerfile
    container_name: grpcui-service
    command: -port $GRPCUI_PORT -plaintext scheduler-service:${SCHEDULER_PORT}
    restart: always
    ports:
      - ${GRPCUI_PORT}:${GRPCUI_PORT}
    depends_on:
      - scheduler-service

Add GRPCUI_PORT=5000 to compose .env file. (.env file beside docker-compose)

We will return to this service later.

Scheduler service: Golang

Our goal is to develop a gRPC server with Go. The typical pipeline for developing a gRPC server is quite straightforward. You define your gRPC schema inside a .proto file (see here for more info). Then you compile (Actually you transform) the .proto to your target programming language using a protocol buffer compiler tool and import it to your project. then you use a gRPC framework in your target language to run a gRPC server. This server uses .proto models in the function parameters. Next you can define corresponding database layer models and use a converter to transform between them. You receive gRPC models vis gRPC server, convert them to database models and store them in a database. In case of queries, you query the data from the database, transform them to gRPC models and return them to the user.

Here is a summary of what we are going to do: We first install protoc in our development environment. Then initialize our go project, define our proto scheme and compile it using the above tool and then run an initial gRPC server. Then we add database layer models and classes.

Create a folder named scheduler-service inside scheduler folder.

Create a Dockerfile inside scheduler-service and set the contents to

FROM golang:1.19

ENV PROTOC_VERSION=23.3
ENV PROTOC_ZIP=protoc-${PROTOC_VERSION}-linux-x86_64.zip
RUN apt-get update && apt-get install -y unzip
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/$PROTOC_ZIP \
    && unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
    && unzip -o $PROTOC_ZIP -d /usr/local 'include/*' \ 
    && rm -f $PROTOC_ZIP
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
RUN go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
RUN export PATH="$PATH:$(go env GOPATH)/bin"

WORKDIR /usr/src/app

Create a directory called "keys" within the scheduler-service folder and create a file named access_token.public.pem. Then go to here and generate an rsa 256 Key-Pair. Copy the public key to access_token.public.pem and keep the private key somewhere (We will use it later). This file acts as the public key of the auth service. Auth service generate an RSA256 key-pair and sign the JWTs with the private key. Other services verify the signature of the JWT using the public key for authentication purposes.

Add the following to the service part of our docker-compose.yml file.

  scheduler-service:
    build: 
      context: ./scheduler-service
      dockerfile: Dockerfile
    container_name: scheduler-service
    command: sleep infinity
    environment:
      DATABASE_SCHEME: mongodb
      DATABASE_DOMAIN: scheduler-db-service
      DATABASE_PORT: ${MONGODB_PORT}
      DATABASE_USER_FILE: /run/secrets/scheduler-db-user
      DATABASE_PASS_FILE: /run/secrets/scheduler-db-pass
      DATABASE_DB_NAME_FILE: /run/secrets/scheduler-db-dbname
      KAFKA_BROKERS: kafka1-service:${KAFKA1_PORT}
      AUTH_PUBLIC_KEY_FILE: /run/secrets/auth-public-key
      # TOPICS_FILE: ''
      ENVIRONMENT: development
      SERVER_PORT: ${SCHEDULER_PORT}
    ports:
      - ${SCHEDULER_PORT}:${SCHEDULER_PORT}
    volumes:
      - ./scheduler-service:/usr/src/app
    secrets:
      - scheduler-db-user
      - scheduler-db-pass
      - scheduler-db-dbname
      - auth-public-key

...

secrets:
  # this is temorary for development environment. 
  auth-public-key:
    file: scheduler-service/keys/access_token.public.pem

We are going to do all the development inside a docker container without installing Golang in our host machine. To do so, we run the containers and then attach VSCode to the scheduler-service container. As you may noticed, the Dockerfile for scheduler-service has no entry-point therefore we set the command value of scheduler-service to sleep infinity to keep the container awake.

Now run docker-compose up -d --build

While running, attach to the scheduler service by clicking bottom-left icon and then select attach to running container. Select scheduler-service and wait for a new instance of VSCode to start. At the beginning the VScode asks us to open a folder inside the container. We have selected WORKDIR /usr/src/app inside our Dockerfile, so we will open this folder inside the container. This folder is mounted to scheduler-service folder inside the host machine using docker compose volume, therefor whatever change we made will be synced to the host folder too.

After opening the folder /usr/src/app, open a new terminal and initialize the go project by running go mod init github.com/<your_username>/play-microservices/scheduler/scheduler-service. This command will create a go.mod file.

Run go get -u google.golang.org/grpc. This is a gRPC framework for running grpc server using Golang.

Run go get -u google.golang.org/grpc/reflection. We add reflection to our gRPC server so that our grpcui-service can connect to it and retrieve the endpoints and messages easily for debugging purposes.

Now create a folder named proto and create a file named job.proto inside. Set the content from here

Run protoc --go_out=./proto --go-grpc_out=./proto proto/*.proto. This command compile our .proto file to Golang. Two files will be created. job.pb.go and job_grpc.pb.go. The first contains the proto models and the second contains the code for job service interface (We need to create our service and implement this interface).

Note: We adopt a Golang project structure that aligns with the recommended guidelines stated here

Create a folder named config and a file named config.go. Set the contents from here. Also create a file named .env in the same folder. we will put our internal environment variables here. Set contents from here. Create another file named .env.topics for putting kafka topics. For production environment, we pass the topics file via docker compose secrets and send the position of file via TOPICS_FILE environment variable. Then we load the contents of the file.

Create a folder named pkg in the root directory (beside mod.go). We will put general packages here. Inside Create a folder named logger then a file named logger.go and set the contents from here.

Create this folder tree: internal/models/job/grpc. inside grpc folder create a file named job_service.go and set the contents to

package grpc

import (
    context "context"
    proto "github.com/<your_username>/play-microservices/scheduler/scheduler-service/proto"

    codes "google.golang.org/grpc/codes"
    status "google.golang.org/grpc/status"
)

type JobService struct {
proto.UnimplementedJobsServiceServer
}

func NewJobService() *JobService {
    return &JobService{}
}

func (j *JobService) CreateJob(ctx context.Context, req *proto.CreateJobRequest) (*proto.CreateJobResponse, error) {
    return nil, status.Errorf(codes.Unimplemented, "method CreateJob not implemented")
}

func (JobService) GetJob(context.Context, *proto.GetJobRequest) (*proto.GetJobResponse, error) {
    return nil, status.Errorf(codes.Unimplemented, "method GetJob not implemented")
}

func (JobService) ListJobs(context.Context, *proto.ListJobsRequest) (*proto.ListJobsResponse, error) {
    return nil, status.Errorf(codes.Unimplemented, "method ListJobs not implemented")
}

func (JobService) UpdateJob(context.Context, *proto.UpdateJobRequest) (*proto.UpdateJobResponse, error) {
    return nil, status.Errorf(codes.Unimplemented, "method UpdateJob not implemented")
}

func (JobService) DeleteJob(context.Context, *proto.DeleteJobRequest) (*proto.DeleteJobResponse, error) {
    return nil, status.Errorf(codes.Unimplemented, "method DeleteJob not implemented")
}

Create a folder named server inside internal folder. Then a file named server.go. set the content to

package server

import (
    "log"
    "net"

    "github.com/<your_username>/play-microservices/scheduler/scheduler-service/config"
"github.com/<your_username>/play-microservices/scheduler/scheduler-service/pkg/logger"
    MyJobGRPCService "github.com/<your_username>/play-microservices/scheduler/scheduler-service/internal/models/job/grpc"
    JobGRPCServiceProto "github.com/<your_username>/play-microservices/scheduler/scheduler-service/proto"

    "google.golang.org/grpc"
    "google.golang.org/grpc/reflection"
)

type server struct {
        log       logger.Logger
    cfg       *config.Config
}

// NewServer constructor
func NewServer(log logger.Logger, cfg *config.Config) *server {
    return &server{log: log, cfg: cfg}
}

func (s *server) Run() error {
    lis, err := net.Listen("tcp", ":"+s.cfg.ServerPort)
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }

    grpc_server := grpc.NewServer()

    job_service := MyJobGRPCService.NewJobService()
    JobGRPCServiceProto.RegisterJobServiceServer(grpc_server, job_service)
    reflection.Register(grpc_server)

    log.Printf("server listening at %v", lis.Addr())
    if err := grpc_server.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
        return err
    }

    return nil
}

Create a folder named cmd and a file named main.go. Set the content to

package main

import (
    "log"

    "github.com/<your_username>/play-microservices/scheduler/scheduler-service/config"
"github.com/<your_username>/play-microservices/scheduler/scheduler-service/pkg/logger"
    "github.com/<your_username>/play-microservices/scheduler/scheduler-service/internal/server"
)

func main() {

    cfg, err := config.InitConfig()
    if err != nil {
        log.Fatal(err)
    }

    appLogger := logger.NewApiLogger(cfg)
    appLogger.InitLogger()
    appLogger.Info("Starting user server")
    appLogger.Infof(
        "AppVersion: %s, LogLevel: %s, Environment: %s",
        cfg.AppVersion,
        cfg.Logger_Level,
        cfg.Environment,
    )
    appLogger.Infof("Success parsed config: %#v", cfg.AppVersion)

    s := server.NewServer(appLogger, cfg)
    s.Run()
}

Run go mod tidy

Run go run cmd/main.go

While our server is running, go to docker desktop and restart grpcui-service. Now go to http://localhost:5000/. If everything goes on plan, you can connect to the server.

Invoking any of the methods results in method xxx not implemented because we still have not implemented our job service methods.

return to VSCode instance that is already attached to our scheduler-service. Stop the service by hitting ctl + c

Create a file named job.go inside models folder. Inside this file we are going to define database layer models corresponding to proto models. Logic for converting from/to proto models comes here. Also we define an interface for the database of our job model and an interface for the event messaging of our job model. Set the contents of the job.go from here.

// databas interface for Job model
type JobDB interface {
    Create(ctx context.Context, job *Job) (*Job, error)
    Update(ctx context.Context, job *Job) (*Job, error)
    GetByID(ctx context.Context, jobID primitive.ObjectID) (*Job, error)
    DeleteByID(ctx context.Context, jobID primitive.ObjectID) (error)
    GetByScheduledKey(ctx context.Context, jobScheduledKey int) (*Job, error)
    DeleteByScheduledKey(ctx context.Context, jobScheduledKey int) (error)
    ListALL(ctx context.Context, pagination *utils.Pagination) (*JobsList, error)
}

//Message broker interface for Job model
type JobsProducer interface {
    PublishCreate(ctx context.Context, job *Job) error
    PublishUpdate(ctx context.Context, job *Job) error
    PublishRun(ctx context.Context, job *Job) error
}

Create a folder named validation inside job folder. Then a file named validate.go. Set the contents from here. Inside this file we put the logic for validation of our input models.

We have defined two interfaces for database and message broker operations. We then pass these two interfaces to our JobService model and do our logic there without knowing which database engine or messaging broker we are using. This gives us the flexibility to select whatever database (mongo or postgres) or message broker (kafka or rabitmq) we want.

For scheduling we use this package. Run go get github.com/reugn/go-quartz/quartz

Now change the definition of JobService inside job_service.go: Final file is here

type JobService struct {
    jobDB         models.JobDB
    jobsProducer  models.JobsProducer
    jobsScheduler scheduler.Scheduler
        proto.JobsServiceServer
}

func NewJobService(jobDB models.JobDB, jobsProducer models.JobsProducer, jobsScheduler scheduler.Scheduler) *JobService {
    return &JobService{jobDB: jobDB, jobsProducer: jobsProducer, jobsScheduler: jobsScheduler}
}

Now it is time to select a database engine and implement JobDB interface using it. Run go get go.mongodb.org/mongo-driver. Create a folder named database inside internal/models/job folder. Then a file named job_db_mongo.go. Set the contents from here

Create a folder named mongodb inside pkg directory and then a file named mongodb.go. This package is used to initialize our mongo db database. Set the contents from here.

Now we select a message broker framework and implement JobsProducer interface using it. Run go get github.com/segmentio/kafka-go. Create a folder named message_broker inside internal/models/job folder. Then create a filed named job_producer_kafka.go here we implement JobsProducer interface. Set the contents from here.

Create a folder named kafka inside pkg and then a file named kafka.go. This package is used to initialize kafka connection. Set the contents from here.

Some notes on kafka architecture:

Topics: Core abstraction in kafka and represent stream of records.

Partitions: Topics which are a chain of records can be divided to partitions to enable parallel processing and scalability. Partitions can be distributed among brokers or reside only in one broker.

Brokers: Brokers are the Kafka servers that form the cluster. They store and manage the published records in a distributed manner.

Replication: Kafka provides replication of data for fault tolerance. Each partition can have multiple replicas spread across different brokers. Replicas ensure that if a broker fails, another broker can take over and continue serving the data seamlessly.

Producers: Producers are responsible for publishing data to Kafka topics.

Consumers: Consumers read data from Kafka topics. Consumers belongs to consumers groups. Each consumer can subscribe to multiple topics but only for one partition of a topic.

Configuring a kafka environment can be tricky and depends on use cases.

Now we reiterate through main.go, server.go and job_service.go files and complete the contents. Change the contents of main.go from here and server.go from here to include mongo database and kafka brokers. We initialize them inside main and pass them to the server struct.

Add the remaining packages from pkg folder. Install the required packages via go get <packagename> command.

"github.com/pkg/errors"

Set the contents for job_service.go from here. The code for the CreateJob function is as follows:

func (js *JobService) CreateJob(ctx context.Context, req *proto.CreateJobRequest) (*proto.CreateJobResponse, error) {
    js.log.Infof("JobService.CreateJob: grpc message arrived : %v", req)
    v := validator.CreateJobRequestValidator{CreateJobRequest: req}
    v_err := v.Validate()
    if v_err != nil {
        js.log.Errorf("JobService.CreateJob: input validation failed for request : %v with error %v", req, v_err.Error())
        return nil, status.Errorf(codes.InvalidArgument, "Invalid request body.")
    }

    job, err0 := models.JobFromProto_CreateJobRequest(req)
    if err0 != nil {
        js.log.Errorf("JobService.CreateJob: cannot create proto request")
        return nil, status.Errorf(codes.InvalidArgument, "Invalid request body.")
    }
    js.log.Infof("JobService.CreateJob: proto converted to db model : %v", job)
    job.Status = int32(proto.JobStatus_JOB_STATUS_SCHEDULED)
    jobFingerPrint := fmt.Sprintf("%s:%s:%s:%p", job.Name, job.Description, job.JobData, &job.ScheduleTime)
    job.ScheduledKey = int(fnv1a.HashString64(jobFingerPrint)) //We assume 64 bit systems!
    //store the job in the database
    js.log.Infof("JobService.CreateJob: Creating job in the database")
    created, err := js.jobDB.Create(ctx, job)
    if err != nil {
        js.log.Errorf("JobService.CreateJob: %v", err.Error())
        return nil, status.Errorf(codes.Internal, "Cannot save job. db error")
    }

    jobID := created.Id.Hex()
    js.scheduleJob(ctx, job.ScheduledKey, jobID, job.ScheduleTime)
    js.jobsProducer.PublishCreate(ctx, created)

    return &proto.CreateJobResponse{Id: jobID}, nil
}

We receive createJob in our gRPC server. We first convert .proto model to database layer model in job := models.JobFromProto_CreateJobRequest(req). Then we set the status to SCHEDULED and save it to the database and retrieve the id. We then schedule the job and publish the topic-job-create to be consumed by other services like reports service and finally we return the CreateJobResponse to the user. Inside the schedule function, We retrieve the job from database, then set the jobState to RUNNING and save it again to the database. Then we publish topic-job-run to be consumed by our job runner service. We then listen to topic-job-run-result event which will be triggered by job runner. If the result was success we change the state of our job to COMPLETE save it to the database.

Run go mod tidy

Run go run cmd/main.go

Now go to docker desktop an restart grpcui service. Then go to http://localhost:5000/. If everything goes according to plan, you can connect to the service. Select CreateJob from method name and fill in the form. For Schedule time if you pass a time before time.Now() the schedule will trigger immediately. For job data depending on the job type we need to send a specific Json string. For email type, we need to send a json with the following structure:

{
    "SourceAddress": "example@example.com",
    "DestinationAddress": "example@example.com",
    "Subject": "Message From example@example.com contact form",
    "Message": "This is a test!!!!"
}

Push Invoke button. You will receive created job id.

Go to http://localhost:8081/ and check the database.

Now go to http://localhost:8080/. You can see that 3 topics have been created. The number of messages for topic-job-create and topic-job-run is 1. Because we have published topic-job-create once and topic-job-run has been published inside our scheduled function.

Stop the service by hitting ctl + c

A note on kafka topics creation: There are generally two common approaches for topic creation in a microservice architecture.

Centralized Topic Creation: a dedicated team or infrastructure administrators are responsible for creating and managing Kafka topics.

Self-Service Topic Creation: In this approach, each microservice is responsible for creating and managing its own Kafka topics. Here, In the development environment we follow this approach and the producer of a topic is responsible for topic creation. We define topic names in .env.topic file for development environment and load them from that file. For production environment, we can receive the file path for the topics using TOPICS_FILE environment variables (location of the file from docker compose secrets) and then load the data from it.

We need to listen to topic-job-run results. To accomplish this we have to subscribe for topic-job-run-result topic. To accomplish that, create a file named job_consumer_kafka.go in the path internal/models/job/message_broker. Then set the contents from here.

Add the following to server.go file.

jobsConsumer := kafka.NewJobsConsumerGroup(s.log, job_db)
jobsConsumer.Run(ctx, cancel, s.kafkaConn, s.cfg)

Run go mod tidy then run go run cmd/main.go

Go to http://localhost:8080/ and under consumers you can see job-run-result-consumer-scheduler which is the group id of our consumer. Now go to Topics-> topic-job-create and in the messages tab click the preview of the value of the message and copy the json structure of the message. Something like this (You need to copy yours because when listening, we search the database for job by the Id):

{
    "Id": "64c07994fcf987a39313754f",
    "Name": "n1",
    "Description": "s1",
    "ScheduleTime": "1970-01-01T00:00:00Z",
    "CreatedAt": "2023-07-26T01:40:36.888841Z",
    "UpdatedAt": "2023-07-26T01:40:36.8888412Z",
    "Status": 2,
    "JobType": 0,
    "JobData": "d1",
    "ScheduledKey": 2534671878703447000
}

Set the value of job status of json string to 4.

Now go to topics -> topic-job-run-result and click produce message from top right corner. Paste the json string into value and click on produce message. If everything goes according to plan, you can see the results from the terminal of VSCode. Also you can go to mongo express at http://localhost:8081/ and see that job status has changed to 4.

Now lets add authentication to our grpc server. Create a folder named interceptors inside internal folder. then a file named auth_interceptor.go. Set the contents from here.

Change the grpc_server creation inside server.go to match the following:

auth_interceptor := Interceptors.NewAuthInterceptor(s.log,s.cfg)
    grpc_server := grpc.NewServer(
        grpc.UnaryInterceptor(auth_interceptor.AuthInterceptor),
    )

Run go run cmd/main.go. Now an auth interceptor in active and protect our grpc server. If you invoke any method, the response would be Missing Authorization header

go to this site and put your generated rsa key-pairs in the corresponding boxes. Then define a simple jwt like this:

{
  "iss": "microservice-scheduler.com",
  "sub": "sheniqua",
  "aud": "maxine",
  "iat": 1689904636,
  "exp": 1689910603,
  "role": "admin"
}

Set other configs as shown in the following image and then click the left arrow. Copy the encoded token to the clipboard.

Now go to docker desktop and restart grpcui-service then go to http://localhost:5000/. For each request you make, set a metadata with name= authorization and the value= <token_copied>

This time when you invoke the methods, authentication will pass. happy debugging :)

To DO

Add tests

Add tracing using Jaeger

Add monitoring and analysis using grafana

Refactoring

I would love to hear your thoughts. Please comment on your opinions. If you found this helpful, let's stay connected on Twitter! xaledhosseini.

Play Microservices: Authentication service

Khaled Hosseini — Tue, 20 Jun 2023 05:06:39 +0000

This is the 2nd part of a series of articles under the name "Play Microservices". Links to other parts:
Part 1: Play Microservices: Bird's eye view
Part 2: You are here
Part 3: Microservices: Scheduler
Part 4: Play Microservices: Email service
Part 5: Play Microservices: Report service
Part 6: Play Microservices: Api-gateway service
Part 7: Play Microservices: Client service
Part 8: Play Microservices: Integration via docker-compose
Part 9: Play Microservices: Security

The source code for the project can be found here:

Our goal is to create an authentication service for our microservices application. To achieve authentication, we require three separate services: a database service, a cache service, and the authentication api service. In the development environment, we include three additional services for debugging purposes. These services are pgadmin to manage our database service, redis-commander service to manage our cache service and grpcui service to test our gRPC api.

At the end, the project directory structure will appear as follows:

Tools

The tools required In the host machine:

Docker: Containerization tool
VSCode: Code editing tool
Dev containsers extension for VSCode
Docker extension for VSCode
Git

The tools and technologies that we will use Inside containers for each service:

Database service: Postgres
Cache service: Redis
pgadmin service: Pgadmin
Redis-commander service: Rediscommander
grpcui service: grpcui for testing our gRPC server.
Auth api service:
- Rust: Programming language
- Tonic: gRPC framework for Rust
- Diesel: Query builder and ORM for our database communication.
- Redis : For our redis server communications.

Docker dev environment

When developing inside a Docker container, you only need to install Docker, Visual Studio Code, and the Dev Containers and Dockerextensions on VS Code. Then you can run a container using Docker and map a host folder to a folder inside the container, then attach VSCode to the running container and start coding, and all changes will be reflected in the host folder. If you remove the images and containers, you only need to create the container using the Dockerfile again and copy the contents of the host folder to the container folder to start again. Under the hood, When attaching VSCode to a running container, Visual Studio code install and run a special server inside the container which handle the sync of changes between the container and the host machine.

Database service: postgres

Create a folder for the project and choose a name for it (such as 'microservice'). Then create a folder named auth. This folder is the root directory of authentication service. You can then open the root folder in VS Code by right-clicking on the folder and selecting 'Open with Code'.

Inside the root directory create a folder with the name auth-db-service, then create the following files inside.

Create a Dockerfile and set content to FROM postgres:15.3

Create a file named pass.txt and set content to a password

Create a file named user.txt and set content to admin

Create a file named db.txt and set content to users_db

Create a file named url.txt and set content to auth-db-service:5432:users_db:admin:password123

Create a file named .env in the root directory and set the content to AUTH_POSTGRES_PORT=5432.

Inside root directory create a file named docker-compose.yml and add the following content.

version: '3'
services:
# database service for auth service
  auth-db-service:
    build:
      context: ./auth-db-service
      dockerfile: Dockerfile
    container_name: auth-db-service
    secrets:
      - auth-db-user
      - auth-db-pass
      - auth-db-db
      - auth-db-url
    environment:
      POSTGRES_USER_FILE: /run/secrets/auth-db-user
      POSTGRES_PASSWORD_FILE: /run/secrets/auth-db-pass
      POSTGRES_DB_FILE: /run/secrets/auth-db-db
    ports:
      - '${AUTH_POSTGRES_PORT}:${AUTH_POSTGRES_PORT}'
    volumes:
      -  auth-db-service-VL:/var/lib/postgresql/data 

volumes:
  auth-db-service-VL:

secrets:
  auth-db-user:
    file: auth-db-service/user.txt
  auth-db-pass:
    file: auth-db-service/pass.txt
  auth-db-db:
    file: auth-db-service/db.txt
  auth-db-url:
    file: auth-db-service/url.txt

In our Docker Compose file, we use secrets to securely share credential data between containers. While we could use an .env file and environment variables, this is not considered safe. When defining secrets in the Compose file, Docker creates a file inside each container under the /run/secrets/ path, which the containers can then read and use. For example, we have set the path of the Docker Compose secret auth-db-pass to the POSTGRES_PASSWORD_FILEenvironment variable. We will be using these secrets in other services later in the project.

pgAdmin service

The purpose of this service is solely for debugging and management of our running database server in the development environment.

Inside root directory create a folder with the name pgadmin-service

Create a Dockerfile and set content to FROM dpage/pgadmin4:7.3

Create a file named .env file and set the content to

PGADMIN_DEFAULT_EMAIL=admin@admin.com
PGADMIN_DEFAULT_PASSWORD=password123

Create a file named servers.json and set the content to

{
    "Servers": {
      "1": {
        "Name": "auth-db-service",
        "Group": "Servers",
        "Host": "auth-db-service",
        "Port": 5432,
        "MaintenanceDB": "users_db",
        "Username": "admin",
        "PassFile": "/run/secrets/auth-db-url",
        "SSLMode": "prefer"
      }
    }
  }

Add the following lines to the .env file of the docker-compose (the .env file at the root directory of the project.)

PGADMIN_CONTAINER_PORT=80
PGADMIN__PORT=5050

Add the following to the service part of the docker-compose.yml.

  pgadmin:
    build:
      context: ./pgadmin-service
      dockerfile: Dockerfile
    container_name: pgadmin-service
# load pgadmin defauld username and password to the environment.
    env_file:
      - ./pgadmin-service/.env
    ports:
      - "${PGADMIN__PORT}:${PGADMIN_CONTAINER_PORT}" 
    secrets:
      - auth-db-url
    volumes:
      - ./pgadmin-service/servers.json:/pgadmin4/servers.json #
    depends_on:
      - auth-db-service

Now open a terminal in your project directory and run docker-compose up. Docker Compose will download and cache the required images before starting your containers. For the first run, this may take a couple of minutes. If everything goes according to plan, you can then access the pgAdmin panel at http://localhost:5050/ and log in using the pgAdmin credentials from the .env file inside the pgadmin-service container. Once inside pgAdmin, you should see that it has successfully connected to the auth-db-service container. You can also register any other running PostgreSQL services and connect to them directly. However, in this case, you will need to provide the credentials for the specific database service in order to connect.

Now run docker-compose down

Cache service: Redis

Inside root directory create a folder with the name auth-cache-service

Create a Dockerfile and set content to FROM redis:7.0.11-alpine

Create a file named pass.txt and set content to a password

Create a file named users.acl and set content to user default on >password123 ~* &* +@all

Create a file named redis.conf and set content to aclfile /run/secrets/auth-redis-acl

Create a file named .env and set the content to REDIS_URI_SCHEME=redis #rediss for tls.

Add AUTH_REDIS_PORT=6379 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  auth-cache-service:
    build:
      context: ./auth-cache-service
      dockerfile: Dockerfile
    container_name: auth-cache-service
    secrets:
      - auth-redis-pass
      - auth-redis-acl
    command: redis-server /usr/local/etc/redis/redis.conf
    ports:
      - '${AUTH_REDIS_PORT}:${AUTH_REDIS_PORT}'
    volumes:
      - auth-cache-serviceVL:/var/vcap/store/redis
      - ./auth-cache-service/redis.conf:/usr/local/etc/redis/redis.conf

Add the following to the secrets part of the docker-compose.yml.

  auth-redis-pass:
    file: auth-cache-service/pass.txt
  auth-redis-acl:
    file: auth-cache-service/users.acl

Here, we first define users.acl for credentials. Then create a redis.conf file inside which we determine the aclfile path. We are going to pass the .acl file via docker compose secrets, therefore the path would be /run/secrets/auth-redis-acl. Then from docker compose file using command redis-server /usr/local/etc/redis/redis.conf we apply the config file (which has already mounted via a volume). Now run docker-compose up again. If everything goes according to plan, you can connect to redis terminal inside the container and check the user. To do so, click on bottom-left corner (Dev container) and select Attach to running container.

To attach to the auth-cache-service, go to the list of running containers and select auth-cache-service. This will launch a new instance of Visual Studio Code attached to that service. Within the newly launched instance, open a new terminal and type redis-cli to connect to the Redis command line interface.
Once connected, running the ping command will result in an error message that says "NOAUTH Authentication required" because our server is password-protected. To address this, run the AUTH password command (note: password refers to the password that you defined inside the users.acl and pass.txt files within the auth-cache-service folder). Running ping again after authenticating should return a response of pong.

run docker-compose down

Redis commander service

This service exists only in the development environment for debugging purposes. We use it to connect to auth-cache-service and manage the data.

Inside root directory create a folder with the name auth-redis-commander-service

Create a Dockerfile and set content to FROM rediscommander/redis-commander:latest

Add Auth_REDIS_COMMANDER_PORT=8081 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Add the following to the service part of the docker-compose.yml.

  auth-redis-commander-service:
    build:
      context: ./auth-redis-commander-service
      dockerfile: Dockerfile
    container_name: auth-redis-commander-service
    hostname: redis-commander-service
    # restart: always
    secrets:
      - auth-redis-pass
    ports:
      - ${Auth_REDIS_COMMANDER_PORT}:${Auth_REDIS_COMMANDER_PORT}
    environment:
      REDIS_HOST: auth-cache-service
      REDIS_PORT: ${AUTH_REDIS_PORT}
      REDIS_PASSWORD_FILE: /run/secrets/auth-redis-pass
    volumes:
      - redisCommanderVL:/data
    depends_on:
      - auth-cache-service

Here we passed the REDIS_PASSWORD_FILE via secrets.

Now run docker-compose up. this will runs the containers. Now go to http://localhost:8081/ and voila! the commander has attached to our redis server.

As you can see, you can view and add data to the redis server.

run docker-compose down

Auth-grpcui-service

Before we dive into coding our auth-server, let's start by initializing a gRPC UI service. This service provide an easy to use ui to test our grpc service in development environment.

Create a folder named Auth-grpcui-service inside root folder.

Create a Docker file and set contents to FROM fullstorydev/grpcui:v1.3.1

Add the following to the services part of the docker-compose.yml file.

  auth-grpcui-service:
    build:
      context: ./auth-grpcui-service
      dockerfile: Dockerfile
    container_name: auth-grpcui-service
    command: -port $AUTH_GRPCUI_PORT -plaintext auth-service:${AUTH_SERVICE_PORT}
    ports:
      - ${AUTH_GRPCUI_PORT}:${AUTH_GRPCUI_PORT}

Our grpc service is ready. We will use it later.

Auth service: Rust

Our goal is to develop a gRPC server with Rust. To achieve this, we begin by defining our protocol buffers (protobuf) schema in a .proto file. We then use a protocol buffer compiler tool to generate model structs in Rust based on this schema. These models are known as contract or business layer models, and they should not necessarily match the exact data structures stored in the database. Therefore, it's recommended to separate the models of the gRPC layer from the models of the database layer, and use a converter to transform between them.
This approach has several benefits. For instance, if we need to modify the database models, there is no need to change the gRPC layer models as they are decoupled. Furthermore, by using object-relational mapping (ORM) tools, we can more easily communicate with the database within our code.

In summary, to build a gRPC app using Rust with a database, we first define our protocol buffers schema in a .proto file and use tonic to generate the gRPC side code. Once the gRPC side is ready, we use diesel to define database schema and run the migration files against our database. Afterward, we define our database models with Diesel.
The next step is tying these two components(gRPC and Diesel models) together. For that, we create a converter class that can translate our gRPC tonic models into Diesel models. This allows us to move data to and from our database. We then use our Diesel ORM to communicate with the database from both the server and clients in our gRPC app. lets start.

create a folder named auth-service inside root directory.

create a Dockerfile and set the contents to the following code (Note: You can install rust at host machine first, then init a starter project and then dockerize your app. But here we do all the things inside a container without installing rust on the host machine):

FROM rust:1.70.0 AS base
ENV PROTOC_VERSION=23.3
ENV PROTOC_ZIP=protoc-${PROTOC_VERSION}-linux-x86_64.zip
RUN apt-get update && apt-get install -y unzip && apt-get install libpq-dev
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/$PROTOC_ZIP \
    && unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
    && unzip -o $PROTOC_ZIP -d /usr/local 'include/*' \ 
    && rm -f $PROTOC_ZIP
RUN apt install -y netcat
RUN cargo install diesel_cli --no-default-features --features postgres

We install protobuf and libpq-dev on our container. The first is used by tonic-build to compile .proto files to rust and the second is required by diesel-cli for database operations.

Add AUTH_SERVICE_PORT=9001 to the .env file of the docker-compose (the .env file at the root directory of the project.)

Create a file named .env inside auth-service and set the contents from here. We will use this variables later, but we need to have it in this step because we have declared it inside docker-compose file.

Create a folder named keys inside auth-service. Inside this folder we put RSA keys for jwt generation and signing. Copy the files from here. You can Create your own rsa256 key pairs from here.

Add the following to the service part of the docker-compose.yml.

  authentication-service:
    build:
      context: ./auth-service
      dockerfile: Dockerfile
      # target: developement
    # restart: always
    # command: bash -c 'while ! nc -z auth-db-service $AUTH_POSTGRES_PORT; do echo ..; sleep 1; done; echo backend is up;cargo sqlx database create; cargo run --bin migrate; cargo run --bin server;'
    container_name: authentication-service
    command: sleep infinity
    environment:
      DATABSE_SCHEME: postgresql
      DATABSE_DOMAIN: auth-db-service
      DATABSE_PORT: ${AUTH_POSTGRES_PORT}
      DATABSE_USER_FILE: /run/secrets/auth-db-user
      DATABSE_PASSWORD_FILE: /run/secrets/auth-db-pass
      DATABSE_DB_FILE: /run/secrets/auth-db-db
      REDIS_SCHEME: redis # rediss
      REDIS_DOMAIN: auth-cache-service
      REDIS_PORT: ${AUTH_REDIS_PORT}
      REDIS_PASS_FILE: /run/secrets/auth-redis-pass
      SERVER_PORT: ${AUTH_SERVICE_PORT}
      REFRESH_TOKEN_PRIVATE_KEY_FILE: /run/secrets/auth-refresh-private-key
      REFRESH_TOKEN_PUBLIC_KEY_FILE: /run/secrets/auth-refresh-public-key
      ACCESS_TOKEN_PRIVATE_KEY_FILE: /run/secrets/auth-access-private-key
      ACCESS_TOKEN_PUBLIC_KEY_FILE: /run/secrets/auth-access-public-key
    secrets:
      - auth-db-user
      - auth-db-pass
      - auth-db-db
      - auth-redis-pass
      - auth-access-public-key
      - auth-access-private-key
      - auth-refresh-public-key
      - auth-refresh-private-key
    env_file:
      - ./auth-service/.env
      - ./auth-cache-service/.env
    # you can access the database service using browser from: AUTH_POSTGRES_PORT and from within docker: AUTH_POSTGRES_CONTAINER_PORT
    ports:
      - ${AUTH_SERVICE_PORT}:${AUTH_SERVICE_PORT}
    volumes:
      - ./auth-service:/usr/src/app 
    depends_on:
      - auth-db-service
      - auth-cache-service

Add the following to the secrets section of the docker compose file.

  auth-access-public-key:
    file: auth-service/keys/access_token.public.pem
  auth-access-private-key:
    file: auth-service/keys/access_token.private.pem
  auth-refresh-public-key:
    file: auth-service/keys/refresh_token.public.pem
  auth-refresh-private-key:
    file: auth-service/keys/refresh_token.private.pem

Now run docker-compose up -d --build we start our containers in detach mode. we temporarily set the command for the auth-service as sleep infinity to keep the container alive. Now we will do all the development inside the container.

Click on bottom-left icon and select Attach to running container and then select auth-service. A new instance of VSCode will open. On the left side, it asks you to open a folder. We need to open the /usr/src/app inside the container. A folder which is mapped to our auth-service folder via docker-compose volume. Now Our VSCode is running inside the /usr/src/app folder inside the container. Then whatever change we made on this folder will sync to our auth-service folder inside the host machine.

Open a new terminal inside new VSCode instance that is attached to auth-service container. run cargo init . This will initialize our rust app.

run cargo add tonic --features "tls". Tonic is a gRPC server.

run cargo add tonic-reflection. We add reflection to our tonic server. Reflection provides information about publicly-accessible gRPC services on a server, and assists clients at runtime to construct RPC requests and responses without precompiled service information.

Create a folder named proto inside auth-service (VSCode has already attached to the app directory inside container, which is mapped to auth-service folder inside host), then create a file named user.proto and define the proto schema inside it. For the contents of the file see here

We will use tonic on top of Rust to deal with gRPC, so we need to add tonic-build dependency in the Cargo.toml. To do that run: cargo add --build tonic-build and cargo add prost. Tonic build uses prost to compile .proto files. Create an empty folder named proto inside src folder (We will set the tonic output folder to this folder so that tonic create compiled .proto files here) and then create a file named Build.rs inside auth-service folder add paste the following code:

use std::{env,path::PathBuf};
fn main()->Result<(),Box<dyn std::error::Error>>{
   env::set_var("OUT_DIR", "./src/proto");
   let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());
   tonic_build::configure()
   .build_server(true)
   .out_dir(out_dir.clone())
   .file_descriptor_set_path(out_dir.join("user_descriptor.bin"))
   .compile(&["./proto/user.proto"],&["proto"])//.compile(&["../proto/user.proto"],&["../proto"])//first argument is .proto file path, second argument is the folder.
   .unwrap_or_else(|e| panic!("protobuf compile error: {}", e));

   Ok(())
}

Create models/user/grpc folder tree inside src and then create a file inside named user_grpc_service.rs. Add the following code:

use tonic::{Request, Response, Status};
use crate::proto::{
    user_service_server::UserService, CreateUserResponse, CreateUserRequest,
    LoginUserRequest, LoginUserResponse,
    RefreshTokenRequest,RefreshTokenResponse, LogOutRequest, LogOutResponse
};

#[derive(Debug, Default)]
pub struct MyUserService{}

#[tonic::async_trait]
impl UserService for MyUserService{
    async fn create_user(
        &self,
        request: Request<CreateUserRequest>,
    ) -> Result<Response<CreateUserResponse>, Status> {
       println!("Got a request: {:#?}", &request);
       let reply = CreateUserResponse { message: format!("Successfull user registration"),};
       Ok(Response::new(reply))
    }

    async fn login_user(
        &self,
        request: Request<LoginUserRequest>,
    ) -> Result<Response<LoginUserResponse>, Status> {

        println!("Got a request: {:#?}", &request);

        let reply = LoginUserResponse {
            access_token : "".to_string(),
            access_token_age : 1,
            refresh_token : "".to_string(),
            refresh_token_age : 1,
        };

        Ok(Response::new(reply))
    }

    async fn refresh_access_token(
        &self,
        request: Request<RefreshTokenRequest>,
    ) -> Result<Response<RefreshTokenResponse>, Status> {
        println!("Got a request: {:#?}", &request);

        let reply = RefreshTokenResponse {
            access_token : "".to_string(),
            access_token_age : 1,
        };

        Ok(Response::new(reply))
    }

    async fn log_out_user(
        &self,
        request: Request<LogOutRequest>,
    ) -> Result<Response<LogOutResponse>, Status> {
        println!("Got a request: {:#?}", &request);

        let reply = LogOutResponse {
            message : "Logout successfull".to_string()
        };

        Ok(Response::new(reply))
    }
}

Create another file inside grpc folder named user_profile_grpc_service.rs and set the contents to:

use tonic::{Request, Response, Status};
use crate::proto::{
    user_profile_service_server::UserProfileService, GetUserResponse, GetUserRequest, ListUsersRequest, ListUsersResponse
};

#[derive(Debug, Default)]
pub struct MyUserProfileService{}

#[tonic::async_trait]
impl UserProfileService for MyUserProfileService{
    async fn get_user(&self, request: Request<GetUserRequest>) -> Result<Response<GetUserResponse>, Status>{
        println!("Got a request: {:#?}", &request);
        let reply = GetUserResponse {
            id: 1,
            name: "name".to_string(),
            email: "email".to_string(),
        };
        Ok(Response::new(reply))
    }
   async fn list_users(&self, request: Request<ListUsersRequest>) -> Result<Response<ListUsersResponse>, Status> {
     return Err(Status::internal("unimplemented."))
  }
}

Inside user folder (parent of grpc folder) create a file named grpc.rs and set the contents to:

pub mod user_grpc_service;
pub mod user_profile_grpc_service;

pub use crate::models::user::grpc::user_grpc_service::MyUserService;
pub use crate::models::user::grpc::user_profile_grpc_service::MyUserProfileService;

inside models folder (parent of src folder) create a file named user.rs and set the contents to:

pub mod grpc;

inside src folder (parent of models), create a file named models.rs and set the content to

pub mod user;

Some notes on how gRPC is initialized via .proto files: To define a schema for our gRPC service, we create a user.proto file. Inside this file, we define a service called UserService, which serves as an interface with functions that have their own input and output parameters. When we run a protocol buffer compiler (no matter the target programming language), it generates this service as an interface, which Rust implements as a trait. To complete our gRPC service implementation, we need to define our own service class and implement the generated trait. This involves creating a structure that will serve as a gRPC service class and adding method implementations that correspond to the services defined in the UserService trait.

We have created a .env file for auth-service. This file contains our environment variables like private and public keys ages.

Run cargo add dotenv.

Create a file named config.rs inside src folder. In this file, we are going to read environment variables along with the secrets that are passed via docker-compose. See file contents here. We will use this module later inside code to manage environment variables.

Run cargo add tokio --features "macros sync rt-multi-thread" Tokio is a runtime for writing reliable asynchronous applications with Rust.

add the following code to the main.rs

mod models;
mod config;
mod proto {
    include!("./proto/proto.rs");
    pub(crate) const FILE_DESCRIPTOR_SET: &[u8] = include_bytes!("./proto/user_descriptor.bin");
}

use models::user::grpc::MyUserService;
use models::user::grpc::MyUserProfileService;

use tonic::{transport::Server};
use proto::user_service_server::UserServiceServer;
use proto::user_profile_service_server::UserProfileServiceServer;


use config::Config;
use std::net::SocketAddr;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config::init();

    let mut server_addr = "[::0]:".to_string();
    server_addr.push_str(&config.server_port);
    let addr: SocketAddr = server_addr.as_str()
    .parse()
    .expect("Unable to parse socket address");

    let user_service = MyUserService::default();
    let user_profile_service = MyUserProfileService::default();
    let reflection_service = 
    tonic_reflection::server::Builder::configure()
.register_encoded_file_descriptor_set(proto::FILE_DESCRIPTOR_SET)
        .build()
        .unwrap();


    Server::builder()
        .add_service(UserServiceServer::new(user_service))
        .add_service(UserProfileServiceServer::new(user_profile_service ))
        .add_service(reflection_service)
        .serve(addr)
        .await?;
    Ok(())
}

Now our gRPC part is ready. Run cargo run. If everything goes according to plan, our service will start. Go to docker desktop and restart auth-grpcui-service service. Now go to http://localhost:5000/ and voila. You can test the server.

Return back to VSCode and stop the server (ctl + c).

Next step is to prepare our database models and migrations. For this we need a running database server and the database_url to connect to it. For development environments it is common to pass this type of data via environment variables. Here we are following an approach that is more common in production environment and pass credentials via docker-compose secrets and then pass the location of the secrets via environment variables. If you run printenv you can see the list of environment variables. We can make the database_url using the environment variables provided to us. Actually, Inside config.rs, we generate database url using environment variables received via docker-compose.

        let database_scheme = get_env_var("DATABSE_SCHEME");
        let database_domain = get_env_var("DATABSE_DOMAIN");
        let database_port = get_env_var("DATABSE_PORT");
        let database_name_file = get_env_var("DATABSE_DB_FILE");
        let database_name = get_file_contents(&database_name_file);
        let database_user_file = get_env_var("DATABSE_USER_FILE");
        let database_user = get_file_contents(&database_user_file);
        let database_pass_file = get_env_var("DATABSE_PASSWORD_FILE");
        let database_pass = get_file_contents(&database_pass_file);
        //database url for postgres-> postgresql://[user[:password]@][netloc][:port][/dbname][?param1=value1&...]
        let database_url = format!("{}://{}:{}@{}:{}/{}",
        database_scheme,
        database_user,
        database_pass,
        database_domain,
        database_port,
        database_name);

In order to make database_url available in the terminal (to be used by diesel cli), we need to create it from the environment values we received via docker-compose. To do so, create a file named db_url_make.sh in the auth-service directory (beside cargo.toml). Then add the following code:

#!/bin/bash

db_name_file="${DATABSE_DB_FILE}"
db_name=$(cat "$db_name_file")
db_user_file="${DATABSE_USER_FILE}"
db_user=$(cat "$db_user_file")
db_pass_file="${DATABSE_PASSWORD_FILE}"
db_pass=$(cat "$db_pass_file")

db_scheme="${DATABSE_SCHEME}"
db_domain="${DATABSE_DOMAIN}"
db_port="${DATABSE_PORT}"

db_url="${db_scheme}://${db_user}:${db_pass}@${db_domain}:${db_port}/${db_name}"
echo "database url is ${db_url}"
export DATABASE_URL=${db_url}

Now run source db_url_make.sh then you can check if the DATABASE_URL available via printenv command.

If you encountered errors like command not found it is possibly due to the difference between end of line characters in linux and windows. You can convert the new lines in VSCode from bottom right corner.

Run cargo add diesel --features "postgres r2d2" Diesel is ORM and query builder. r2d2 feature gives us connection pool management capabilities.

Run cargo add diesel-derive-enum --features "postgres" This package helps us use rust enums directly with diesel ORM. (At the moment diesel doesn't support native rust enums).

Run diesel setup this will create a folder named migrations and diesel.toml file beside cargo.toml.

Run diesel migration generate create_users This command creates two files named XXX_create_users/up.sql and XXX_create_users/down.sql. Edit up.sql to:

CREATE TYPE role_type AS ENUM (
  'admin',
  'customer'
);

CREATE TABLE users (
        id int NOT NULL PRIMARY KEY DEFAULT 1,
        name VARCHAR(100) NOT NULL,
        email VARCHAR(100) NOT NULL UNIQUE,
        verified BOOLEAN NOT NULL DEFAULT FALSE,
        password VARCHAR(100) NOT NULL,
        role role_type NOT NULL DEFAULT 'customer'
    );
CREATE INDEX users_email_idx ON users (email);

Edit down.sql to

DROP TABLE IF EXISTS "users";
DROP TYPE role_type;

Run diesel migration run this will run the migrations against database_url and creates schema.rs inside src folder. In case of error relation "users" already exists, you can run diesel print-schema > src/schema.rs to create the schema file for you. Also if you want to revert the migration just run: diesel migration revert, This will apply down.sql file to the database. you can run diesel migration run again.

We did the migrations using cli. For production environments, we can run the migrations on start inside the code. To accomplish this, Run cargo add diesel_migrations and then add the following codes to the main.rs

use diesel::{pg::PgConnection, prelude::*};
use diesel_migrations::EmbeddedMigrations;
use diesel_migrations::{embed_migrations, MigrationHarness};
pub const MIGRATIONS: EmbeddedMigrations = embed_migrations!("./migrations");

...

// add this to the start of main function.
 let config = Config::init();

        let database_url = config.database_url.clone();
    let mut connection = PgConnection::establish(&database_url)
    .unwrap_or_else(|_| panic!("Error connecting to {}", database_url));

    _ = connection.run_pending_migrations(MIGRATIONS);

Now run cargo run This will runs the migrations against the database on app start. You can go to pgadmin panel and see users table under servers: auth-db-service: databases: users_db: schema: public: tables.

Create a folder named db inside src/models/user/ and then a file named model.rs inside db folder.

//Our ORM models.
use diesel::prelude::*;

#[derive(Queryable, Selectable, PartialEq,  Identifiable, Debug,)]
#[diesel(table_name = crate::schema::users)]
#[diesel(check_for_backend(diesel::pg::Pg))]
pub struct User {
    pub id: i32,
    pub name: String,
    pub email: String,
    pub verified: bool,
    pub password: String,
    pub role: RoleType,
}

#[derive(Insertable, Queryable, Debug, PartialEq)]
#[diesel(table_name = crate::schema::users)]
#[diesel(check_for_backend(diesel::pg::Pg))]
pub struct NewUser {
    pub name: String,
    pub email: String,
    pub password: String,
    pub role: RoleType,
}

#[derive(diesel_derive_enum::DbEnum,Debug,PartialEq)]
#[ExistingTypePath = "crate::schema::sql_types::RoleType"]
#[DbValueStyle = "snake_case"]//Important. it should be styled according to .sql: see https://docs.rs/diesel-derive-enum/2.1.0/diesel_derive_enum/derive.DbEnum.html
pub enum RoleType {
    ADMIN,
    CUSTOMER
}

inside db.rs, We define our database models. To clarify a bit more, We have grpc layer models that we have defined in our .proto file. Necessarily we do not save the exact same models in our database. One reason is the difference of data types and another is we may have our logic in the database layer. Inside this file, we will implement the logic to convert to/from proto layer models. Rust made this conversion easy to implement. We use fromand intotraits in rust. add the following code to the model.rs file.

use crate::proto::{ CreateUserRequest, CreateUserResponse, GetUserResponse,LogOutResponse, RoleType as RoleTypeEnumProto};
...
impl std::convert::From<RoleTypeEnumProto> for RoleType {
    fn from(proto_form: RoleTypeEnumProto) -> Self {
        match proto_form {
            RoleTypeEnumProto::RoletypeAdmin => RoleType::ADMIN,
            RoleTypeEnumProto::RoletypeCustomer => RoleType::CUSTOMER,
        }
    }
}

impl std::convert::From<i32> for RoleTypeEnumProto {
    fn from(proto_form: i32) -> Self {
        match proto_form {
            0 => RoleTypeEnumProto::RoletypeAdmin,
            1 => RoleTypeEnumProto::RoletypeCustomer,
            _ => RoleTypeEnumProto::RoletypeCustomer,
        }
    }
}

//Convert from User ORM model to UserReply proto model
impl From<User> for GetUserResponse{
    fn from(rust_form: User) -> Self {
        GetUserResponse{
            id: rust_form.id,
            name: rust_form.name,
            email: rust_form.email
        }
    }
}

//convert from CreateUserRequest proto model to  NewUser ORM model
impl From<CreateUserRequest> for NewUser {
    fn from(proto_form: CreateUserRequest) -> Self {
        //CreateUserRequest.role is i32
        let role: RoleTypeEnumProto = proto_form.role.into();
        NewUser {
            name: proto_form.name,
            email: proto_form.email,
            password: proto_form.password,
            role: role.into(),
        }
    }
}

impl From<&str> for LogOutResponse {
    fn from(str: &str)-> Self {
        LogOutResponse{
            message: str.to_string()
        }
    }
}
impl From<&str> for CreateUserResponse {
    fn from(str: &str)-> Self {
        CreateUserResponse{
            message: str.to_string()
        }
    }
}

Another logic we may consider inside the model.rs file is to define some interfaces for database and cache operations. This interfaces will be used by our grpc services without linking to a specific database or cache service. add the following code to the model.rs file.

use std::error::Error;
...
#[tonic::async_trait]
pub trait UserDBInterface {
    async fn is_user_exists_with_email(&self,email: &String)-> Result<bool, Box<dyn Error>>;
    async fn insert_new_user(&self,new_user: &NewUser)-> Result<usize, Box<dyn Error>>;
    async fn get_user_by_email(&self,email: &String)-> Result<std::option::Option<User>, Box<dyn Error>>;
    async fn get_user_by_id(&self,id: &i32)-> Result<User, Box<dyn Error>>;
    async fn get_users_list(&self,page: &i64, size: &i64)-> Result<Vec<User>, Box<dyn Error>>;
}

#[tonic::async_trait]
pub trait UserCacheInterface {
    async fn set_expiration(&self,key: &String,value: &String, seconds: usize)-> Result<(), Box<dyn Error>>;
    async fn get_value(&self,key: &String)-> Result<String, Box<dyn Error>>;
    async fn delete_value_for_key(&self,keys: Vec<String>)-> Result<String, Box<dyn Error>>;
}

add the following line to db.rs (inside user folder).

pub use crate::models::user::db::model::{User,NewUser,RoleType,UserDBInterface, UserCacheInterface};

Another logic to be implemented inside db folder is the code for validation of our input models. Create a file named validation.rs inside db folder and set the contents from here. Add the following code to db.rs inside user folder.

pub mod validation;
pub use crate::models::user::db::validation::Validate;

Now it is time to Prepare our grpc services using UserDBInterfaceand UserCacheInterface. But before we need to add some packages for our cache service and manipulation of JWTs.

Run cargo add redis --features "tokio-comp".

Run cargo add argon2 we use it for password hashing.

Run cargo add serde --features derive serde is a framework for serializing and deserializing Rust data structures.

Run cargo add uuid --features "serde v4"

Run cargo add chrono --features serde

Run cargo add jsonwebtoken jsonwebtoken Create and parses JWT.

Run cargo add log env_logger

Most of the above packages are being used for JWT generation and validations. JWT is a standard way for securely transmitting information between parties. The schematic pipeline of how typically JWT is used across services can be illustrated as below:

Another module which is required here is one for creation and verifying json web tokens (JWT). Create a folder named utils inside src and then a file named jwtoken.rs inside utils folder and set the contents from here.

Create a file named utils.rs inside src and set the contents to pub mod jwtoken;

Change the MyUserService definition to match the following code.

use crate::models::user::db::{UserDBInterface, UserCacheInterface};
...
pub struct MyUserService {
    pub db: Box<dyn UserDBInterface + Send + Sync>,
    pub cache: Box<dyn UserCacheInterface + Send + Sync>,
    pub env: Config,
}

We will using above interface in our grpc methods without knowing which database engine or cache service we are using. for example, for create_user function we have:

        async fn create_user(
        &self,
        request: Request<CreateUserRequest>
    ) -> Result<Response<CreateUserResponse>, Status> {
        info!("MyUserService: create_user Got a request: {:#?}", &request);
        let create_user_request: CreateUserRequest = request.into_inner();
        match create_user_request.validate() {
            Ok(_)=>{
                info!("MyUserService:create_user Input validation passed for CreateUserRequest")
            }
            Err(err)=> {
                error!("MyUserService:create_user Input validation failed for CreateUserRequest. {:#?}",&err);
                return Err(Status::invalid_argument("invalid argument"));
            }
        }

       match self.db.is_user_exists_with_email(&create_user_request.email).await {
            Ok(exists) => {
                if exists {
                    //https://docs.rs/tonic/latest/tonic/struct.Status.html
                    return Err(Status::already_exists("User with that email already exists"));
                }
            },
            Err(err) => {
                error!("MyUserService:create_user Error checking email existence. check the database. {:#?}",&err);
                return Err(Status::internal("Problem checking user"));
            }
        };

        //we do not save the password. but a hash of it.
        let salt = SaltString::generate(&mut OsRng);
        let hashed_password = Argon2::default()
        .hash_password(create_user_request.password.as_bytes(), &salt)
        .expect("Error while hashing password")
        .to_string();

        let mut new_user: NewUser = create_user_request.into();
        new_user.password = hashed_password;

        info!("Inserting new user: {:#?}", &new_user);

        match self.db.insert_new_user(&new_user).await {
            Ok(_)=> {
                let reply: CreateUserResponse = "Successfull user registration".into();
                return Ok(Response::new(reply))
            },
            Err(err)=> {
                error!("Problem creating user with error: {:#?}",&err);
                return Err(Status::internal("Error registering user."));
            }
        };
    }

The complete code for user_grpc_service.rs and user_profile_grpc_service.rs can be found here and here respectively.

Now it is time to prepare a database and a cache engine so that we can pass them to the constructor of MyUserService and MyUserProfileService structs.

inside models/user/db folder create a file named diesel_postgres.rs and set the contents from here. then another file named redis.rs and then set the contents from here.

Add the following to the db.rs file inside user folder.

pub mod diesel_postgres;
pub mod redis;
pub use crate::models::user::db::diesel_postgres::{PostgressDB,PgPool,PgPooledConnection};
pub use crate::models::user::db::redis::RedisCache;

Change the contents of main.rs to

extern crate log;
// in the main file, declare all modules of sibling files.
mod config;
mod models;
mod schema;
mod utils;
mod proto {
    include!("./proto/proto.rs");
    pub(crate) const FILE_DESCRIPTOR_SET: &[u8] = include_bytes!("./proto/user_descriptor.bin");
}

//proto
use models::user::grpc::MyUserService;
use models::user::grpc::MyUserProfileService;
use tonic::transport::Server;
use proto::user_service_server::UserServiceServer;
use proto::user_profile_service_server::UserProfileServiceServer;

pub use crate::models::user::db::{PostgressDB,PgPool,PgPooledConnection};
pub use crate::models::user::db::RedisCache;
//diesel
use diesel::{pg::PgConnection, prelude::*};
use diesel::r2d2::{ Pool, ConnectionManager };
use diesel_migrations::EmbeddedMigrations;
use diesel_migrations::{embed_migrations, MigrationHarness};
pub const MIGRATIONS: EmbeddedMigrations = embed_migrations!("./migrations");

use config::Config;
use std::net::SocketAddr;


#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {

    // env::set_var("RUST_BACKTRACE", "full");
    env_logger::init();

    let config = Config::init();
    let redis_url = config.redis_url.clone();
    let database_url = config.database_url.clone();
    let server_port = config.server_port.clone();
    let redis_client = match redis::Client::open(redis_url) {
        Ok(client) => {
            println!("✅Connection to the redis is successful!");
            client
        }
        Err(e) => {
            println!("🔥 Error connecting to Redis: {}", e);
            std::process::exit(1);
        }
    };

    let manager = ConnectionManager::<PgConnection>::new(&database_url);
    let db_pool = match Pool::builder().build(manager) {
        Ok(pool) => {
            println!("✅ Successfully created connection pool");
            pool
        }
        Err(e) => {
            println!("🔥 Error creating connection pool: {}", e);
            std::process::exit(1);
        }
    };

    //do migration

    let mut connection = PgConnection::establish(&database_url)
    .unwrap_or_else(|_| panic!("Error connecting to {}", database_url));
    _ = connection.run_pending_migrations(MIGRATIONS);

    //user service
    let my_user_service = MyUserService { 
        db: Box::new(PostgressDB { db_pool: db_pool.clone() }),
        cache: Box::new(RedisCache{redis_client: redis_client.clone()}),
        env: config.clone()
    };
    let user_service = UserServiceServer::new(my_user_service);

    let my_user_profile_service = MyUserProfileService {
        db: Box::new(PostgressDB { db_pool: db_pool })
    };
    let profile_service = UserProfileServiceServer::new(my_user_profile_service);

    //reflection service for user
    let reflection_service = tonic_reflection::server::Builder::configure()
        .register_encoded_file_descriptor_set(proto::FILE_DESCRIPTOR_SET)
        .build()
        .unwrap();

    let mut server_addr = "[::0]:".to_string();
    server_addr.push_str(&server_port);
    let addr: SocketAddr = server_addr.as_str()
    .parse()
    .expect("Unable to parse socket address");



    let svr = Server::builder()
        .add_service(profile_service)
        .add_service(user_service)
        .add_service(reflection_service)
        .serve(addr);
    println!("Server listening on {}", addr);
    svr.await?;
    Ok(())
}

Run cargo run If everything goes according to plan, our app will be compiled and starts. Go to docker desktop and restart auth-grpcui-service service. Assuming everything proceeds as expected, auth-grpcui-service will start at http://localhost:5000/

As we have used reflector for our service, grpcui can automatically retrieve the endpoints and parameters automatically.

lets test our server. Select CreateUser from method name and fill in the form, then click Invoke. As long as nothing unexpected happens, we receive "message": "Successfull user registration" to check, go to pgadmin and inspect user's table.

Now select loginUser and fill in the form with the data You used for CreateUser. You will receive access and refresh tokens. Now you can go to the redis-commander-service at http://localhost:8081/ and see the cache data.

Stop the server by hitting ctl + c We have defined two separate services in our .proto file. User service and profile service. We need to do authentication for profile service. (user services messages already contain credentials). To do so, add the following packages:

Run cargo add futures

Run cargo add tower

Run cargo add hyper

Inside src, add a folder named interceptors and the a file named auth.rs. Set the content from here

Create a file named interceptors.rs inside src and set the content to pub mod auth;

Add he following code to the main function:

...
mod interceptors;

//interceptor
use crate::interceptors::auth::AuthenticatedService;
...
let profile_service_intercepted = AuthenticatedService {
        inner: profile_service,
        env: config,
};
...
let svr = Server::builder()
        .add_service(profile_service_intercepted)
        .add_service(user_service)
        .add_service(reflection_service)
        .serve(addr);

A summary on the auth interceptor: Requests for UserProfileService will go through this interceptor first. Inside this file, we will read the access token, then validate it and if validation passed, we attach the user id and the role of the token to the header of of the request. Inside UserProfileService methods, we read those headers and do the authorization.

Run cargo run. Go to http://localhost:5000/ and do test the app using the following scenario.

Select proto.userService from service name.

Select CreateUser from method name: (name:admin,email:admin@test.com,password: password,role: admin)

Select LoginUser from method name. (email:admin@test.com,password: password)

Copy the value of access_token (Without double quotes)

Select UserProfileService from sevice name.

Select ListUsers from method name. (page:1, size:10)

Add authorizationas key and the token you have copied from the login user as value for metadata. click invoke.

Our service development environment is ready. you can continue coding and debugging.

One important note: for packages, we have used cargo add package_name, and this is not safe, as the packages may change in the future. In case of falling into a dependency hell, here is the package versions inside my cargo.toml file:

To DO

Add tests

Add tracing using Jaeger

Add monitoring and analysis using grafana

Refactoring

I would love to hear your thoughts. Please comment on your opinions. If you found this helpful, let's stay connected on Twitter! xaledhosseini.

Play Microservices: A Bird's-eye view

Khaled Hosseini — Tue, 20 Jun 2023 05:06:20 +0000

Welcome, everyone! This is the first article in a series called "Play Microservices" where I will be implementing a microservices architecture featuring a variety of services and technologies. Throughout this series, I aim to provide concise explanations without being overly verbose. So without further ado let's begin.

First:

As I am not going to get into the details of Microservices architecture, I suggest you to have a glance at microservices.io. I aim to introduce the steps involved in developing a typical microservices software; however, please note that the code and steps detailed throughout this series should not be considered complete nor of production-level quality.

The source code for the project can be found here:

Goal application
DevOps review
Applying DevOps principles to our application: Development-> plan
- Architecture selection
- Microsystems architecture design of our app
- Microsystem patterns
- Choreography diagram
- DevOps diagram
- Tools and technologies
- Next

Our goal application:
A simple task scheduling app with authentication. Users register (Admin or user) and log in to the app. Admins can get the list of all users and schedule email jobs to them. Also admins can query for reports.

DevOps review
Before delving into the specifics, we must first take a broad overview of our product, tools and technologies, and development pipeline. In order to produce a final product, companies must progress through a multi-stage pipeline, each stage operating seamlessly and continuously. Due to the numerous stages involved, we will only briefly cover those that are relevant to full-stack developers. DevOps is a software development approach that emphasizes collaboration and communication between developers and IT operations to speed up software delivery, increase reliability, and improve customer satisfaction. It involves a wide range of tools and practices, including automation, continuous integration and delivery, and agile development. Usually the lifecycle of DevOps is symbolized in the form of an infinity loop.

In DevOps, we continuously perform the following tasks:

To summarize the tasks performed in each phase, we can roughly state the following.

Continuous development:
- Plan
- Code
- On each commit integration phase is started.
Continuous integration:
- New code review
- Integrate new code into existing code
- Deploy to build server
- Unit testing
- Deploy to test environment
- Feedback
Continuous testing:
- Integration tests
- Component tests
- Contract tests
- E2E tests
- UI/UX tests
- Security tests
- Feedback
Continuous deployment:
- Deploy to stage environment
- Configure and deploy to production environment
Continuous feedback
Continuous monitoring
Continuous Operations

Applying DevOps principles to our application: Development-> plan
Now we want to apply above approach and go through this pipeline step by step.

Architecture selection

The first step in our journey is to choose an architectural pattern for our application. Our goal is to utilize a microsystems architecture; however, another approach is the Monolithic architecture. Each architecture has its own set of strengths and weaknesses, as well as unique features at its core.

Microsystems architecture design of our app

How to design the microsystem architecture for our app? To design the architecture of our Microservice we follow the instructions in assemblage and A pattern language for microservices.

Discovering system operations: According to the description of our goal application, our system operations can be listed as:
- signUpUser()
- signInUser()
- refreshToken()
- getUser()
- getAllUsers()
- scheduleJob()
- getJob()
- deleteJob()
- updateJob()
- listJobs()
- ExecuteJob(JobTypeEmail)
- createReport()
- listReport()
Services and their collaborations: After conducting further investigations and refactoring, we have compiled a list of our services:
- Authentication service
- Job-Scheduler service
- Email-job Executor service
- Report service
- API gateway service
- Client service

We use the C4 model for visualizing our software architecture. According to this model we have 4 layers of abstraction. If you imagine your software as a map, the 1st layer has the top highest view and then you zoom in and in each layer you provide more details.

In case of our app, we can visualize these layers as follows:

System context diagram

Container diagram

Component diagram : We have 6 microservices each have it's own components.
Code diagram: We have 6 microservices each have it's own components. Each component has its own code diagram.

Microsystem patterns

Our application is based on a microservices architecture, and to simplify the design process, we implemented microservices-specific patterns. The following provides an overview of the patterns that were utilized and the ones that should be taken into consideration. Please note that the goal is simply to demonstrate an implementation of the patterns. The fact that we are using these particular patterns does not necessarily mean they are the best fit for our specific use case.

Service collaboration patterns:
- Event sourcing: persist aggregates as a sequence of events (instead of saving objects in database)
- CQRS: We define a view database (a new service) which subscribe to events from other services (on data change) and save a view of the desired aggregate that we want. This view is read only and we only query from that service. Reports service uses this pattern to create reports of notification tasks.
- API composition: In this pattern, the result for a specific client request is generated from joining multiple queries to multiple services.
- Saga: In this pattern we design a business transaction as a sequence of local transactions (which occurs in a single service locally, then publish an event). Reverse transactions can occur on failed events. Here task scheduler and notification service use this pattern to update the state of scheduled notifications.
- Domain event
Other microservice patterns:
- Database per service.
- Team per service.
- Code source per service.
Individual service patterns / principles:
- TDD (Test driven development)
- BDD (Behavior driven development)
- SOLID principles: Following these patterns ensures adherence to SOLID principles.

Choreography diagram

When we combine all of these components -- the services, their collaborations and connections, and the data flow -- we can represent the architecture of our app in the form of a diagram like this:

DevOps diagram

We can sketch the DevOps pipeline of our app development as follows:

Tools and technologies

Since our goal is to demonstrate the flexibility of microservices architecture in terms of independence from specific technologies and systems, we will employ various programming languages for different services of the system. The following are the tools and technologies utilized in our application:

Development tools:
- Docker: Containerization.
- Visual studio code: Code editor
- Dev containers: Edit code inside containers
Integration / deployment tools:
- Jenkins. Alternatives: circleci, travis-ci, gitlab ci/cd, AWS CodePipeline, Azure Pipelines
service collaboration / communication technologies:
- Kafka: Message broker. alternatives: rabbitmq, mosquitto, NSQ
- grpc
- Rest
- graphql
monitoring technologies:
- Metrices: prometheus
- Tracing: jaeger
- Logging: Elasticsearch
Auth service
- Rust: Programming language
- postgres: database
- redis: cache
- Tonic: gRPC framework for Rust
- Diesel: Query builder and ORM for our database communication.
- Redis : For our redis server communications in rust.
Scheduler service
- Golang : programming language
- gRPC-GO: gRPC framework for golang
- Mongo driver: Query builder for our database communication.
- Kafka go for our message broker communications from go.
- Quartz for scheduling purposes.
Email job executor service
- Python : programming language
- Kafka-python: For our message broker communications from go.
Reports service
- Python : programming language
- grpc: gRPC framework for python
- Pymongo: Query builder for our database communication.
- Kafka-Python for our message broker communications from python.
- grpcio-tools For compiling .proto files to python.
API gateway service
- Golang : programming language
- gRPC-GO: gRPC framework for golang
- Gin: Is a web framework written in Golang
- gin-swagger for our rest api documentation.
Client service
- Typescript : programming language
- Next.js: Web framework that can be used for developing backend and front-end applications.
- Tailwind css: A utility-first CSS framework
- react-hook-form
- TanStack Query: A asynchronous state management.

We've covered some of the phases of our DevOps pipeline, with a focus on the development: plan stage. Within the context of our microservices architecture, each service can have its own independent code base and unique technical implementation. However, there are logical connections between services, so collaboration between teams is required.

Once the codebase environments for each service have been prepared, we need to set up a CI/CD pipeline using tools such as Jenkins. Teams write their code and tests, push them to the code base, and trigger the CI/CD pipeline. The CI/CD tool will check out the code base (including any dependent repositories), start the build, test, and deployment pipeline, and automatically provide feedback to the team.

Next

In the next part of this series, I'll implement the authentication service.

Links to other parts:
Part 1: You are here
Part 2: Play Microservices: Authentication service
Part 3: Microservices: Scheduler
Part 4: Play Microservices: Email service
Part 5: Play Microservices: Report service
Part 6: Play Microservices: Api-gateway service
Part 7: Play Microservices: Client service
Part 8: Play Microservices: Integration via docker-compose
Part 9: Play Microservices: Security

I would love to hear your thoughts. Please comment on your opinions. If you found this helpful, let's stay connected on Twitter! xaledhosseini.

DEV Community: Khaled Hosseini

Data structures and algorithms in depth: C++, Swift, Python, Java, C#, JavaScript.

Memory

Physical layer

Memory: Virtual layer

Location

Arrangement

Algorithms

Fundamental operations

Fundamental Algorithms

Sorting algorithms

Searching algorithms

Algorithm design techniques

Data structures

Contiguous Memory data structures

Discontiguous Memory data structures

Combination of CM and DCM

Linear data structures

Array

DynamicArray

RingBuffer

LinkedList

FreeList

DoubleLinkedList

CircularLinkedList

CircularDoubleLinkedList

Stack

Stack via DynamicArray

Stack via LinkedList

Stack via deque

Queue

Queue via DoubleLinkedList

Queue via RingBuffer

Queue via Double Stack

Deque as Queue

Deque

Deque via DoubleLinkedList

Deque via Array

PriorityQueue

PriorityQueue via DynamicArray

PriorityQueue via LinkedList

PriorityQueue via Deque

PriorityQueue via BinaryHeap

Associative collections

UnorderedMap or HashTable

OrderedMap via HashTable and LinkedList

OrderedMap via HashTable and DynamicArray

SortedMap via Self Balancing Tree

Set

UnorderedSet

OrderedSet via HashTable and LinkedList

SortedSet via Self Balancing Tree

standard library data structures

C++

Swift

CSharp

Python

Java

Non-Linear data structures

Algorithms

Master typescript types in 7 minutes.

Types in C# and Java

primitive types

Non-primitive types

Types in typescript

primitive types

Non-primitive types

Unions of primitive types / values

Extend object types

Map an existing type to another by applying a mapper type

Filter types by applying a conditional type

Mix classes using mixins

Picking properties from another type

How TLS / mTLS handshakes Work?

TLS/SSL (transport layer security)

mTLS (mutual transport layer security)

Play Microservices: Security

Contents:

Summary

Basics

Extend `object` types

Map an existing type to another by applying a `mapper type`

Filter types by applying a `conditional type`

Mix classes using `mixins`