DEV Community: George Hadjisavva

RabbitMQ for Developers: A Comprehensive Introduction

George Hadjisavva — Thu, 24 Aug 2023 17:36:24 +0000

Introduction

Imagine a bustling train station in the heart of a metropolitan city. Trains arrive and depart, carrying with them thousands of passengers to their respective destinations. This station, with its intricate rail networks and precise scheduling, ensures that every passenger gets to where they need to be, efficiently and on time. In the world of software development, RabbitMQ is akin to this train station, orchestrating the flow of messages between various applications.

RabbitMQ, a leading message broker software, is pivotal in ensuring applications in distributed systems communicate effectively. But what makes it stand out in the crowded world of message brokers? Let's dive in.

What is RabbitMQ?

RabbitMQ is an open-source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). At its core, it allows different parts of a system to send, receive, and hold messages. Think of it as the train tracks and signals that guide the flow of data, just as rails and signals guide trains.

Why Use RabbitMQ?

Scalability: RabbitMQ can handle millions of messages per second, making it perfect for high-throughput systems.

Flexibility: It supports multiple messaging protocols like AMQP, MQTT, STOMP, and more.

Reliability: Messages can be persisted to disk, ensuring they aren't lost.

Distributed Deployment: Supports clustering and high availability configurations.

Core Concepts

Producer: Application that sends the messages.
Consumer: Application that receives the messages.
Queue: Buffer that stores messages sent by the producer.
Exchange: Directs messages to queues using rules called bindings.
Binding: A link between a queue and an exchange.

Types of Exchanges

Direct Exchange: Sends messages to specific queues based on a routing key.
Topic Exchange: Directs messages based on wildcard matches between the routing pattern and the routing key.
Headers Exchange: Uses message header attributes for routing.
Fanout Exchange: Routes messages to all the queues bound to it.

Use Cases

Task Queues: Distribute tasks among multiple workers.
Publish/Subscribe Pattern: Send a message to multiple consumers.
Message Routing: Direct messages based on rules.
Topics: Dynamically filter messages.

Conclusion

RabbitMQ is a powerful tool that every developer should consider adding to their toolkit. Its flexibility, reliability, and scalability make it a prime choice for modern applications that require robust communication mechanisms. As with all tools, the best way to understand it is to try it out and experiment with its features.

Twitter
Newsletter

Unveiling Docker: A Primer on the Revolutionary Containerization Technology

George Hadjisavva — Sat, 20 May 2023 18:46:19 +0000

DevGnosis : https://devgnosis.substack.com/

Docker is a robust platform that leverages lightweight containers, streamlining container management tasks for developers. With Docker, you can effortlessly create and deploy applications, which are analogous to images in traditional virtual machine environments, but tailored for containerization. By handling container provisioning and mitigating networking complexities, Docker simplifies the development process. Moreover, Docker offers a built-in registry functionality, enabling convenient storage and versioning of Docker applications

The Docker application abstraction proves beneficial as it shields us from the underlying technology utilized to implement the service, similar to VM images. We employ Docker applications generated through our service builds and store them within the Docker registry, allowing us to seamlessly proceed with our development process.

Docker can also alleviate some of the downsides of running lots of services locally for dev and test purposes. Rather than using Vagrant to host multiple independent VMs, each one containing its own service, we can host a single VM in Vagrant that runs a Docker instance. We then use Vagrant to set up and tear down the Docker platform itself, and use Docker for fast provisioning of individual services.

How Redis Enhances Microservice Ecosystems: Exploring its Benefits

George Hadjisavva — Sat, 22 Apr 2023 09:52:28 +0000

In the world of microservices, managing data can be a complex task. With multiple services communicating with each other, it's essential to have a system that can handle data storage and retrieval efficiently. Redis is a popular open-source, in-memory data structure store that is used by many companies to solve this problem. Its speed, flexibility, and ease of use make it an ideal choice for microservice ecosystems.

Redis is a key-value-based caching system that can be compared to memcached, but with added features.
It has a schemaless structure that allows for flexibility in defining data tables or schemas.
Redis supports multiple data models and types, making it a versatile database option.
Compared to other database systems, Redis offers advanced features such as sharding, which enables it to withstand concurrent write requests and transactions.
Redis can be used alongside other databases to reduce load and improve performance, or as a primary database based on individual needs.
It is particularly useful in scenarios that require quick data ingestion, data integrity, high efficiency, and replication.

Redis Use-Cases

Redis has a wide range of use cases, making it a versatile tool for various applications. One of the most popular use cases for Redis is caching. By caching frequently accessed data in Redis, microservices can quickly retrieve the data from memory, rather than having to query a database every time. Redis is also used as a message broker, where it can manage queues of messages between services, ensuring reliable and efficient communication. Another popular use case is real-time analytics, where Redis can store and analyze large volumes of data in memory, making it an ideal choice for microservices that require real-time data processing.

Redis as a Caching

Redis is a powerful tool that can be used in a variety of microservice use cases. One common use case is caching, where Redis can significantly improve the performance of microservices by storing frequently accessed data in memory.

Redis in an ecommerce application

In an e-commerce microservice architecture, product pages need to be loaded quickly and efficiently for a smooth user experience. However, querying a database for product information every time a page is requested can be time-consuming and resource-intensive. This is where Redis comes in handy as a caching solution. When a user requests a product page, the microservice can first check if the product information is already available in Redis cache. If it is, the microservice can quickly retrieve the product data from Redis, without having to query the database. This can significantly reduce the response time and improve the performance of the microservice.

Moreover, Redis's in-memory storage model allows for extremely fast read and write operations, making it an ideal choice for caching. Additionally, Redis offers advanced features like expiration time and eviction policies, which allow you to set a timeout for cached data or remove it based on certain criteria. This ensures that the cached data stays fresh and relevant, without taking up unnecessary memory space.

Overall, using Redis to cache product information in an e-commerce microservice can lead to faster response times, improved user experience, and reduced load on the underlying database.

Redis as Message Broker

Message brokering is an important aspect of microservice architectures, as it allows services to communicate with each other efficiently and reliably. Redis can be used as a message broker, where it can manage queues of messages between services. For example, in a ride-hailing app, Redis can be used to manage ride requests and dispatch them to drivers. When a user requests a ride, the ride request can be stored in a Redis queue, which can be monitored by a driver dispatch service. The dispatch service can then pick up the request from the queue and assign a driver to the ride. This way, Redis helps ensure that ride requests are handled efficiently and reliably.

Redis as Real-time analytics

Real-time analytics is another important use case for Redis in microservices. Redis's in-memory storage model allows it to store and analyze large volumes of data in real-time. For example, a social media microservice might use Redis to analyze user behavior in real-time, such as tracking likes, comments, and shares. This data can be stored in Redis and analyzed using Redis's built-in data structures and functions. This can help the microservice make real-time decisions based on user behavior, such as recommending similar posts or content.

In conclusion, Redis is a highly versatile and powerful tool that can greatly benefit microservice ecosystems. Its speed, scalability, and in-memory storage model make it an ideal choice for caching data, managing message queues, and processing real-time analytics. Redis's ability to handle multiple data types and models, and its lack of strict schema requirements, allow for flexible and efficient data management within microservices. Additionally, Redis's advanced features, such as sharding and eviction policies, make it a highly customizable and effective tool for various use cases. Whether used as a primary database or in conjunction with other databases, Redis is a valuable addition to any microservice ecosystem looking to improve performance, efficiency, and user experience.

Subscribe to DevGnosis Newsletter to learn more : https://devgnosis.substack.com/

Motivations for Decomposing Monolithic Applications into Microservices

George Hadjisavva — Tue, 18 Apr 2023 18:47:25 +0000

If you have a monolithic service or application that you want to break down into smaller components, taking an incremental approach is highly recommended. This will allow you to learn about microservices gradually and minimize the risk of making mistakes (which are bound to happen). Instead of trying to tackle the entire monolith at once, think of it as a block of marble that you can chip away at little by little. Trying to blow it up all at once is not a good idea, as it often leads to negative consequences. By taking an incremental approach, you can achieve your goal of breaking down the monolith while avoiding unnecessary risks and complications.

If we plan to break down the monolith gradually, the question arises as to where we should begin. Although we have identified the seams in our application, it is crucial to determine which one should be extracted first. Instead of dividing things arbitrarily, it is wise to consider which part of the codebase will yield the most significant benefits when separated. Therefore, we need to consider some factors that can guide us in selecting the right chisel to start with.

Pace of Change

If we anticipate upcoming changes in our inventory management approach, it could be advantageous to initiate the decomposition process by separating the warehouse seam into a microservice. This approach can enhance our ability to modify the service promptly and effectively since it operates autonomously.

Team Structure

Microservices can bring numerous benefits to a team's structure, promoting autonomy and flexibility. By breaking down the monolithic application into smaller services, developers can take ownership of the services they are responsible for, and can work in smaller teams that focus on specific areas of functionality. This allows teams to be more autonomous and make faster decisions, reducing the need for coordination with other teams.

In addition, microservices can facilitate easier integration of new team members into the project, as they can quickly become familiar with specific services and contribute to their development without having to understand the entire monolith. The modular structure of microservices also makes it easier to reuse code between services and build new services on top of existing ones, providing a more agile and flexible development process.

Microservices can also lead to improved scalability, as individual services can be scaled up or down independently based on their specific usage patterns. This can improve performance and reduce costs, as resources can be allocated more efficiently to meet demand.

Security

Microservices can also provide advantages when it comes to security. Breaking down the monolith into smaller, independent services can reduce the potential impact of security breaches. For instance, if a security breach occurs in one service, the impact will be limited to that service only, and not affect the entire application. This is because each service has its own codebase, data store, and API, which reduces the attack surface area.

Moreover, microservices enable teams to apply different security measures to each service based on its specific requirements. Teams can implement security protocols that are tailored to the specific needs of each service, rather than having a one-size-fits-all approach that may not be suitable for all components of the monolithic application. This can enhance the overall security of the application, as teams can focus on specific vulnerabilities and protect against them more effectively.

In addition, microservices can provide better control over data access and usage. Each service can have its own data store, and teams can implement access controls that limit who can access, modify, and delete the data within each service. This can prevent unauthorized access and ensure that sensitive data is protected.

Technology

Technology stacks play a vital role in the development and deployment of microservices. Microservices architecture allows teams to choose different technologies for each service, enabling them to select the best-suited tool for the specific service's needs. This results in a diverse technology stack for a microservices-based application.

One of the primary technology stacks used in microservices is containerization. Tools like Docker and Kubernetes are used to package, deploy, and manage individual microservices. Containers provide a lightweight and isolated runtime environment for each service, making it easier to deploy and scale services independently.

API gateways are also commonly used in microservices architecture. An API gateway sits between clients and microservices, providing a single entry point for all service requests. API gateways can perform various tasks, such as routing requests to the appropriate service, handling authentication and authorization, and rate-limiting requests to prevent service overload.

Another technology that is often used in microservices is event-driven architecture. This approach involves services communicating with each other via events and messages. Event-driven architecture allows for loosely coupled services and enables asynchronous processing, improving the scalability and resilience of the application.

In addition, cloud platforms are often used in microservices-based applications. Cloud platforms like Amazon Web Services, Microsoft Azure, and Google Cloud provide a range of services and tools that can be used to deploy, manage, and scale microservices. Cloud platforms also offer features like auto-scaling, load balancing, and serverless computing, which can help optimize the performance and cost of microservices-based applications.

Finally, technologies like service discovery, monitoring, and tracing are also important for microservices architecture. Service discovery enables services to locate and communicate with each other, while monitoring and tracing tools allow developers to monitor the performance of individual services and identify issues quickly.

Subscribe to DevGnosis Newsletter to learn more : https://devgnosis.substack.com/

Exploring the RESTful Architecture: How the Web Inspired a New Way of Building APIs

George Hadjisavva — Thu, 06 Apr 2023 17:40:10 +0000

REST, which stands for Representational State Transfer, is an architecture that takes inspiration from the World Wide Web. Although REST comprises numerous principles and limitations, we will concentrate on the ones that are most useful in resolving integration difficulties in the context of microservices and when seeking an alternative interface style to RPC for our services.

The most critical aspect of REST is the notion of resources, which can be thought of as entities that the service is aware of, such as a Customer. The server generates various representations of this Customer upon request, and how a resource appears externally is entirely separated from how it is stored internally. For instance, a client may request a JSON representation of a Customer, even if it is stored in an entirely distinct format. Once a client obtains a representation of this Customer, it may issue requests to modify it, and the server may or may not fulfill them.

Rest And HTTP

The REST style integrates very well with the useful capabilities defined in HTTP. For instance, HTTP verbs like GET, POST, and PUT already have defined meanings in the HTTP specification regarding how they interact with resources. According to the REST architecture, methods should behave consistently on all resources, and luckily, HTTP provides a range of methods we can use. Specifically, GET retrieves a resource idempotently, and POST creates a new resource. By utilizing these HTTP methods, we can eliminate the need for numerous createCustomer or editCustomer methods. Instead, we can simply send a POST request with a customer representation to request the server to create a new resource, and use a GET request to obtain a representation of an existing resource. In these scenarios, we have one endpoint in the form of a Customer resource, and the available operations are integrated into the HTTP protocol.

HTTP offers an extensive range of supportive tools and technologies, such as HTTP caching proxies like Varnish and load balancers like mod_proxy. Additionally, many monitoring tools come equipped with pre-built support for HTTP. These building blocks enable us to handle substantial volumes of HTTP traffic and efficiently route them in a transparent manner. With HTTP, we can also leverage all available security controls to secure our communications, from basic authentication to client certificates. However, it is crucial to use HTTP effectively to reap these benefits. If used poorly, it can be just as insecure and difficult to scale as any other technology. But when used correctly, HTTP provides ample assistance to facilitate our work.

Hypermedia As the Engine of Application State(HATEOAS)

Another principle introduced in REST that can help us avoid the coupling between client and server is the concept of hypermedia as the engine of application state. This is fairly dense wording and a fairly interesting concept, so let’s break it down a bit.

The concept of hypermedia involves including links to various pieces of content in different formats, such as text, images, and sounds, within a particular content piece. This notion is likely familiar to you, as it mirrors how a typical web page functions: by following links, which are a form of hypermedia controls, to view related content. HATEOAS takes this idea further by suggesting that clients should interact with the server (which may trigger state transitions) through links to other resources. Instead of knowing precisely where customers reside on the server and what URI to target, the client should search for and navigate links to locate the necessary information.

Think of the Amazon.com shopping site. The location of the shopping cart has changed over time. The graphic has changed. The link has changed. But as humans we are smart enough to still see a shopping cart, know what it is, and interact with it. We have an understanding of what a shopping cart means, even if the exact form and underlying control used to represent it has changed. We know that if we want to view the cart, this is the control we want to interact with. This is why web pages can change incrementally over time. As long as these implicit contracts between the cus‐ tomer and the website are still met, changes don’t need to be breaking changes.

1.<link rel = “/products” href=”products/item1” />

2.<link rel=”/instantpurchase” href=”instantPurchase/1234” />

In this document, we have two hypermedia controls. The client reading such a docu‐ ment needs to know that a control with a relation of products is where it needs to navigate to get information about the product items, and that instantpurchase is part of the protocol used to purchase the album. The client has to understand the semantics of the API in much the same way as a human being needs to understand that on a shopping website the cart is where the items to be purchased will be.

As a client, I do not require knowledge of the specific URI scheme to purchase the album. All I need to do is access the resource, locate the buy control, and proceed to it. The location of the buy control may change, the URI may undergo modification, or the site may redirect me to an entirely different service, but as a client, I need not worry about these details. This setup provides a significant level of decoupling between the client and server.

Data Formats : JSON or XML

Employing standard textual formats offers clients extensive flexibility when consuming resources, and REST over HTTP enables us to use a diverse range of formats. Although the examples I have illustrated until now employed XML, currently JSON has become a more prevalent content type for services that operate via HTTP.

The fact that JSON is a much simpler format means that consumption is also easier. Some proponents also cite its relative compactness when compared to XML as another winning factor, although this isn’t often a real-world issue.

Despite its popularity, JSON does come with a few drawbacks. While XML defines the link control as a hypermedia control, there is no equivalent standard in JSON. Therefore, custom in-house styles are often utilized to incorporate this concept. To address this issue, the Hypertext Application Language (HAL) has been developed to establish a common set of standards for hyperlinking in JSON (and XML, though it is arguably less necessary for XML). Adhering to the HAL standard allows for the use of tools like the web-based HAL browser to explore hypermedia controls, simplifying the process of creating a client.

Downsides to REST Over HTTP

One potential downside of using REST over HTTP is performance, which may be a concern for services with low-latency requirements. While REST supports alternative formats like JSON or binary, HTTP still introduces overhead for each request. Additionally, REST payloads may not be as lean as those of binary protocols like Thrift.

Although HTTP is suitable for handling large amounts of traffic, it may not be the best option for low-latency communications when compared to other protocols built on Transmission Control Protocol (TCP) or other networking technologies. One such example is WebSockets, which, despite its name, is not necessarily tied to the Web. Once the initial HTTP handshake is completed, WebSockets operate solely as a TCP connection between the client and server, making it a much more efficient way to stream data to a browser. However, it's important to note that using WebSockets doesn't involve much of HTTP or REST.

For server-to-server communications, if extremely low latency or small message size is important, HTTP communications in general may not be a good idea. You may need to pick different underlying protocols, like User Datagram Protocol (UDP), to achieve the performance you want, and many RPC frameworks will quite happily run on top of networking protocols other than TCP.

Subscribe to newsletter for more :
https://architechinsider.substack.com/

Role of Apache Thrift in Twitter's search ranking: Explain how Apache Thrift plays a role in Twitter's search ranking system

George Hadjisavva — Sun, 02 Apr 2023 17:24:41 +0000

Recently, Twitter made waves in the open-source community by publicly sharing its search ranking system, a critical component of the platform's user experience that helps users find the most relevant and timely content based on their search queries. However, the process of ranking tweets in real-time presents a number of technical challenges, such as efficiently processing large volumes of data and communicating between multiple services involved in the ranking process.

Apache Thrift, an open-source framework for communication between services, played a key role in enabling efficient communication between the various services involved in Twitter's search ranking process. By defining data structures and interfaces in a language-agnostic way, Apache Thrift allowed services written in different programming languages to communicate with each other seamlessly and efficiently, making it an ideal choice for a real-time system like Twitter's search ranking.

In this article, we'll explore the role of Apache thrift and specific ways in which Apache Thrift is used , and how it helps solve some of the technical challenges .

What is apache thrift

Apache Thrift is an open-source framework for implementing remote procedure call (RPC) services. It was developed by Facebook in 2007, and later became an Apache Software Foundation project in 2008. The goal of Apache Thrift is to provide a simple and efficient way for services written in different programming languages to communicate with each other over a network.

Thrift works by defining a set of data structures and service interfaces using a domain-specific language called Thrift IDL (Interface Definition Language). This IDL is then used to generate code in multiple programming languages, which can be used to implement the server and client components of the RPC service. Thrift supports a wide range of programming languages, including Java, Python, C++, Ruby, and many others.

Benefits of apache thrift

1.Language Independence: With Thrift, developers can define their services using an interface definition language (IDL), which is independent of any specific programming language. The Thrift compiler then generates code in the target language, allowing developers to write code once and deploy it on multiple platforms.

2.Efficient Data Transfer: Thrift uses a compact binary protocol to transfer data between client and server applications, which reduces network overhead and improves performance. The binary protocol is also designed to be extensible, allowing developers to add custom data types and serialization formats.

3.Cross-Platform Compatibility: Thrift supports a wide range of programming languages, including Java, Python, Ruby, PHP, C++, and many more. This makes it an ideal choice for building applications that need to communicate across different platforms and operating systems.

4.Service Evolution: As applications evolve over time, it is often necessary to add new features and functionality. Thrift makes it easy to add new services without breaking existing clients or servers. This is achieved through versioning, which allows developers to define multiple versions of the same service and choose which version to use at runtime.

5.Scalability: Thrift is designed to be scalable, with support for multiple transport protocols, load balancing, and connection pooling.

How Twitter is utilising Thrift

Twitter uses Thrift to define several data structures for ranking tweets, including ThriftLinearFeatureRankingParams, ThriftAgeDecayRankingParams, ThriftHostQualityParams, and ThriftCardRankingParams. These data structures allow Twitter to store and process tweet ranking information efficiently, resulting in faster and more accurate tweet rankings.

Additionally, Twitter uses Thrift to define a variety of ranking parameters, including score parameters for various tweet features like retweet count, reply count, reputation, and text score. By defining these parameters in a concise and structured format, Twitter can easily modify and experiment with their ranking algorithms without rewriting large amounts of code.

Thrift also allows Twitter to dynamically load custom ranking algorithms and collectors for experimentation purposes. This feature provides Twitter with the flexibility to test new ranking strategies quickly, enabling them to iterate and improve their ranking algorithms continuously.

Here is the link to the Thrift file for the thrift ranking definitions:https://github.com/twitter/the-algorithm/blob/main/src/thrift/com/twitter/search/common/ranking/ranking.thrift

Subscribe to newsletter for more :
https://architechinsider.substack.com/

Optimizing Communication in Microservices: Comparing Synchronous and Asynchronous Methods

George Hadjisavva — Fri, 31 Mar 2023 07:54:40 +0000

When it comes to service collaboration, deciding whether to opt for synchronous or asynchronous communication is one of the most vital choices to make. This decision significantly impacts implementation details.

Synchronous communication necessitates making a call to a remote server that temporarily suspends execution until the operation finishes. Conversely, asynchronous communication enables the caller to return without waiting for the operation to complete and might not place priority on whether or not the operation concludes.

Synchronous communication is often more straightforward to comprehend. We can easily determine if operations have concluded successfully or not. In contrast, asynchronous communication can be highly advantageous for long-running tasks, where it's impractical to keep a connection open between the client and server for an extended period. It's also beneficial for situations where low latency is crucial, and blocking a call while awaiting a response can significantly slow down the system. In the context of mobile networks and devices, firing off requests and presuming that they have succeeded (unless notified otherwise) can ensure that the user interface remains responsive, even if the network is sluggish. However, it's worth noting that handling asynchronous communication may require more complex technology, which we will delve into shortly.

There are two primary modes to consider: synchronous and asynchronous. Each mode can facilitate distinct idiomatic styles of collaboration, either request/response or event-based.

Request/Response Communication (Orchestration)

In the request/response model, a client initiates a request and waits for a response. While this approach is well-suited for synchronous communication, it can also work in the context of asynchronous communication.For instance, a client might launch an operation and register a callback, requesting the server to notify them when the operation is complete.

Event-Based Communication (Choreography)

In an event-based collaboration, we take a different approach. Instead of a client initiating requests for specific tasks to be performed, it informs other parties that a particular event has occurred and expects them to act accordingly. Unlike request/response collaboration, we never instruct anyone else what to do. Event-based systems are inherently asynchronous, and the business logic is more evenly distributed among the various collaborators instead of being centralised in a core system. Furthermore, event-based collaboration enables high decoupling. The client emitting an event has no knowledge of who or what will respond to it, meaning that new subscribers can be added to these events without the client's involvement.

Real life example

Let’s take an example from a real-life banking software , and look at what happens when we create a customer:

A new record is created in the loyalty points bank for the customer.

2.Our postal system sends out a welcome pack.

3.We send a welcome email to the customer.

In terms of implementing the flow, there are two architectural styles to consider. The first is orchestration, which relies on a central system to guide and control the process, similar to a conductor leading an orchestra. The second style is choreography, where each component of the system is informed of its role and allowed to determine the details of its own execution, similar to dancers in a ballet who find their way and respond to others around them.

Let's consider how an orchestration approach could be applied to this flow. In this scenario, the most straightforward solution would be to designate the customer service as the central control point. When a new customer is created, the customer service would communicate with the loyalty points bank, email service, and postal service using a series of request/response calls, as shown in Figure 1.2.

The customer service itself can then track where a customer is in this process. It can check to see if the customer’s account has been set up, or the email sent, or the post delivered.

The downside of the orchestration(request/response) approach is that it can lead to the customer service becoming a centralised governing authority. It can become the focal point of a web, and a central location where the logic is concentrated. I have observed this approach leading to a few intelligent "god" services dictating to weak CRUD-based services.

Choreographed Approach

In an alternative approach, choreography, the customer service could emit an asynchronous event saying "Customer created" instead. The email service, postal service, and loyalty points bank would then simply subscribe to these events and respond accordingly, as shown in Figure 1.3. This method is much more loosely coupled. If another service needed to respond to the creation of a customer, it would simply subscribe to the events and perform its job as required. However, the downside is that the explicit view of the business process depicted in Figure 1.2 is now only implicitly represented in our system.

With that approach implies that there is a need for extra effort to ensure that you can keep an eye on and verify that the necessary tasks have been completed. For instance, would you be able to identify if the loyalty points bank encountered an error and failed to set up the correct account? One strategy that I find useful in addressing this issue is to create a monitoring system that aligns explicitly with the business process view in Figure 1.2. However, this system also tracks the actions of each service as independent entities, enabling you to map any unusual exceptions onto the more explicit process flow.

In my experience, systems that lean towards a event-based (choreographed) approach are typically more flexible, loosely coupled, and adaptable to change. However, additional effort is required to monitor and track the processes across different systems. Conversely, heavily orchestrated implementations can be very fragile and have a higher cost of change. For this reason, I highly recommend striving for a choreographed system, where each service is intelligent enough to comprehend its role in the overall process.

On the other hand synchronous communication is straightforward and provides immediate feedback on the success of a call. However, if we prefer the request/response model but are dealing with processes that take longer, we can initiate asynchronous requests and wait for callbacks. On the contrary, adopting an asynchronous event-based collaboration allows for a choreographed approach, resulting in more decoupled services. This is desirable as it ensures that our services can be released independently of each other.

Subscribe to newsletter for more :
https://architechinsider.substack.com/

Shared Databases in Microservices: A Blessing or a Curse?

George Hadjisavva — Mon, 27 Mar 2023 17:19:23 +0000

The rise of microservices has fundamentally changed the way we approach application development. But with this new architecture comes new challenges, particularly when it comes to database management. One of the key decisions you'll face is whether to use a shared database or separate databases for each microservice. In this article, we'll dive into the shared database dilemma.
Shared Database is by far the most common form of integration that I or any of my colleagues see in the industry is database (DB) integration. In this world, if other services want information from a service, they reach into the database. And if they want to change it, they reach into the database! This is really simple when you first think about it, and is probably the fastest form of integration to start with—which probably explains its popularity.
In the figure below, it can be observed that our registration user interface (UI) creates customers by executing SQL operations directly on the database. The diagram also displays our finance application, which accesses and modifies customer data by running SQL queries on the database. Additionally, the marketing department can update customer information by querying the database.

Initially, we are granting external entities the ability to examine and connect to the internal implementation specifics. The data structures stored in the database are open to all; they are shared in their entirety with any other party that has access to the database. If I choose to modify my schema to enhance the representation of my data or facilitate the maintenance of my system, it is feasible that I may disrupt my consumers(Finance,Registation,Marketing). Essentially, the database serves as an extensive, shared application programming interface (API) that is susceptible to breaking. If I need to modify the procedures linked to, for example, how the Finance administers customers, and this necessitates a change to the database, I must be exceedingly cautious not to impair parts of the schema that are employed by other services. Such a scenario usually mandates a significant amount of regression testing.

Another withdrawn is consumers are tied to specific technology choice . Right now makes sense to store customers in a relational database so my consumers use an appropriate (potentially DB-specific) driver to talk to it . Suppose we determine in the future that a non-relational database would be a superior option for storing data. Would the customer service be able to make that determination? As a result, the consumers are closely linked to the implementation of the customer service, which contradicts our goal of concealing implementation details from consumers to allow the service greater autonomy in modifying its internal workings over time. This effectively eliminates the concept of loose coupling that we previously discussed in previous topics.

Finally , Let's take a moment to consider the behavior aspect. There will be certain rules associated with modifying a customer. Where is this set of rules located? If the consumers are directly manipulating the database, then they are responsible for owning the relevant set of rules. In this scenario, the set of rules for making changes to a customer could be spread across multiple consumers. For example, if the Finance UI, registration UI and Marketing UI all require the ability to modify customer data, then any bug fixes or changes in behavior will need to be implemented in three different places and then deployed. This creates a situation where we are no longer achieving a high degree of cohesion.

However, the integration of a database seems to compromise both of these principles. While database integration facilitates data sharing between services, it fails to address the sharing of behavior. As a result, our internal representation is exposed to consumers, making it challenging to avoid changes that may break the existing functionality, ultimately leading to a sense of fear towards any modification. Therefore, it is advisable to avoid this shared database approach at almost any expense.

Recalling our previous discussion on the fundamental tenets of effective microservices, we emphasized the importance of strong cohesion and loose coupling. However, the integration of a database seems to compromise both of these principles. While database integration facilitates data sharing between services, it fails to address the sharing of behavior. As a result, our internal representation is exposed to consumers, making it challenging to avoid changes that may break the existing functionality, ultimately leading to a sense of fear towards any modification.

Subscribe to newsletter for more :
https://architechinsider.substack.com/

Exploring the Power of ElasticSearch

George Hadjisavva — Thu, 23 Mar 2023 15:15:14 +0000

Elastic Search is a powerful search and analytics engine that is designed to handle large volumes of data and deliver lightning-fast results. It's based on Apache Lucene, a full-text search engine library that is widely used in the industry. Elastic Search adds a distributed layer on top of Lucene, making it scalable, fault-tolerant, and highly available.

One of the key features of Elastic Search is its ability to index and search both structured and unstructured data. Structured data refers to data that is organized into a specific format, such as a database table or a CSV file. Unstructured data, on the other hand, refers to data that is not organized into a specific format, such as text documents, social media posts, or sensor data. Elastic Search can handle both types of data, making it ideal for a wide range of use cases

Capabilities of Elasticsearch

Full Text Search Engine

When it comes to searching through large volumes of data, traditional SQL database management systems are often not equipped to handle the task. That's where Elastic Search comes in as a powerful full-text search engine built on top of Lucene. With Elastic Search, you can perform a wide range of searches, from structured and unstructured data to geo and metric searches, allowing for greater flexibility and precision. Its full-text search capabilities make it an ideal tool for searching through large and complex datasets.

Analytical Engine

While Elastic Search is known for its powerful full-text search capabilities, its analytical use case is even more popular. Elastic Search is frequently used for log analytics and slicing and dicing numerical data, such as application and infrastructure performance metrics. Although Apache Solr was the first to provide faceting, Elastic Search has taken faceting to the next level by allowing users to aggregate data in real-time using its aggregation queries. These queries are instrumental in powering data visualizations across various tools, including Kibana, Grafana, and others. Elastic Search's analytical engine is a key factor in making it a popular choice for businesses that require powerful and flexible data analytics capabilities.

Distributed Architecture Designed for Scaling

Elastic Search was designed to be scalable from the ground up, thanks to its distributed architecture. With Elastic Search, it's possible to scale your system to hundreds of servers and handle petabytes of data.

While distributed systems can be complex, Elastic Search simplifies the process by making many of the scaling decisions automatically and providing a robust management API. Compared to other systems, scaling Elastic Search is much easier, although managing large Elastic Search clusters can be challenging and often requires specialized expertise. Elastic Search also includes automatic data replication to prevent data loss in the event of node failures, further enhancing its reliability and resilience.

Difference between SQL vs ElasticSearch

While SQL databases have been the go-to solution for managing and querying structured data for decades, Elastic Search offers a more flexible approach to data management that can better handle unstructured and semi-structured data. Here are some key differences between SQL databases and Elastic Search:

Data Modeling

SQL databases require a strict schema for data modeling, meaning that the structure of the data needs to be defined in advance. This can make it challenging to handle data that doesn't fit neatly into a structured schema. In contrast, Elastic Search doesn't require a strict schema, allowing for more flexible and dynamic data modeling.

Full-Text Search

While SQL databases can support full-text search, it's often not their primary function, and the performance can be suboptimal for large volumes of data. Elastic Search, on the other hand, was built with full-text search in mind and can efficiently handle large volumes of unstructured data.

Scalability

SQL databases can be scaled horizontally by adding more nodes to a cluster, but this process can be complex and time-consuming. In contrast, Elastic Search is designed to be distributed from the ground up, making it easier to scale horizontally and accommodate petabytes of data.

Performance

Traditional SQL databases require complex joins and aggregations to perform complex queries, which can lead to performance issues when analyzing large data sets in real-time.

In addition Elastic Search's real-time search capabilities allow for the efficient analysis of large and complex data sets, while its flexible data model allows for the storage and retrieval of unstructured and semi-structured data. Elastic Search also includes powerful aggregation capabilities, which enable users to perform complex calculations on large data sets in real-time..

Subscribe to newsletter for Weekly Learnings & News :
https://architechinsider.substack.com/

Why Microservices Matter: Revolutionizing Modern Software Development

George Hadjisavva — Mon, 20 Mar 2023 17:26:54 +0000

In software development, microservices refer to a novel approach to building applications that involves creating small, independent services that collaborate with each other. To better understand this concept, it is important to delve into the specific attributes that distinguish microservices from other software development approaches.

Microservices Introduction

Single Responsibility : Small, and Focused on Doing One Thing Well

As we add new features to our codebase, it inevitably grows larger, making it challenging to identify where changes need to be made. Despite efforts to maintain clear, modular monolithic codebases, the arbitrary in-process boundaries can break down over time. This can result in code related to similar functions being dispersed throughout the codebase, leading to difficulties in bug fixing or implementation.

In a monolithic system, we counteract these challenges by striving for cohesive code, often by creating abstractions or modules. Cohesion, which refers to the grouping of related code, is a crucial concept in microservices. Robert C. Martin's Single Responsibility Principle emphasizes the importance of cohesion, stating that we should "gather together those things that change for the same reason and separate those things that change for different reasons."

Microservices employ a similar approach by using independent services that are designed to align with specific business boundaries, making it clear where the code resides for a particular functionality. By limiting each service to a distinct boundary, we prevent the temptation for it to become overly large and complicated, which can lead to a host of issues.

Autonomous :

Our microservices are designed as separate entities and can be deployed as isolated services on a platform or their own operating system process. We aim to avoid combining multiple services on the same machine to maintain simplicity and ensure that communication between services is through network calls. Services must be able to change independently without affecting consumers, and we need to consider what services should expose and what should be hidden to maintain autonomy. To ensure technology-agnostic Application programming interface (API), we may need to pick the right technology that doesn't constrain technology choices. Decoupling is vital for our microservices, and any changes must be deployable without changing anything else to achieve the benefits of this approach.

Key benefits

Technology Heterogeneity

In a system with multiple collaborating services, we have the flexibility to use different technologies for each service. This approach allows us to select the most suitable tool for each task, rather than being limited to a standardized, one-size-fits-all approach that can compromise performance.

If we need to enhance the performance of a specific component in our system, we may opt to use a different technology stack that can better meet the required performance levels. We might also decide to store data differently for different parts of our system. For instance, for a social network, we could store user interactions(friends) in a graph-oriented database to capture the intricate network of connections. However, we might choose to store user posts in a relational data store, resulting in a heterogeneous architecture, as depicted in the diagram.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fbbxtbg8g5bvlkq41hw7.png)

Microservices also enable us to quickly adopt new technologies and understand how they can benefit us. In traditional monolithic applications, one of the biggest challenges in experimenting with new technology is the associated risks. Even small changes can have significant impacts on the entire system. However, with microservices, we can introduce new programming languages, databases, or frameworks without affecting the entire system, thus reducing the risks involved in technology adoption.

Resilience

In a microservices architecture, service boundaries act as bulkheads that help to isolate problems in case of a component failure. This means that if one service fails, it doesn't cause a cascade effect, and the rest of the system can continue working. On the other hand, in a monolithic service, the failure of one component can bring down the entire system. Although a monolithic system can run on multiple machines to reduce the risk of failure, a microservices architecture can handle service failures and degrade functionality accordingly.

Scaling

In a monolithic service, scaling is an all-or-nothing affair, meaning that we have to scale the entire system, even if only a small part of it is experiencing performance issues. This constraint can lead to unnecessary costs and complexity.

With smaller services, we can just scale those services that need scaling, allowing us to run other parts of the system on smaller, less powerful hardware .

As illustrated in the figure below, we have chosen to scale the post's service and chat service in our social media platform due to the higher traffic volume on those services.

Easy of deployment

Imagine changing one line to a monolithic application with hundreds of lines of code , as you guessed this required to re-deploy the whole application. That could be a larger impact . Regrettably, this implies that our modifications accumulate over time, and we keep piling up changes until the new version of our application that is released to production has an overwhelming number of changes. The greater the difference between releases, the more likely we are to make mistakes, increasing the risk involved.

By using microservices, we can modify and deploy a single service without affecting the rest of the system, enabling us to deploy our code more rapidly. If there is an issue, it can be isolated to a single service, allowing us to quickly roll back. This approach also enables us to provide new functionality to customers more quickly.

Organizational Alignment

Usually large teams and large codebases attract organisational problems . These problems can be exacerbated when the team is distributed. We also know that smaller teams working on smaller codebases tend to be more productive. Microservices allow us to better align our architecture to our organization, helping us minimize the number of people working on any one codebase to hit the sweet spot of team size and productivity. We can also shift ownership of services between teams to try to keep people working on one service colocated.

Subscribe to newsletter for more :
https://architechinsider.substack.com/

Javascript Namespace pattern

George Hadjisavva — Mon, 29 Nov 2021 15:57:12 +0000

Namespace Pattern

Namespace can dramatically reduce the number of globals required and at the same time prevents the collisions or excessive name prefixing .
Its important to know that javascript doesn't have namespaces built into the language syntax , but you can achieve this feature quite easy .Instead of adding functions,objects and variables into global scope you can create one global object and add all the functionality

Refactor anti-pattern to Namespace example

Consider this example

//anti-pattern example
function Read() {}
function Speak() {}
var topic_to_learn = "Javascript";
//objects
var book1 = {}
book1.data = {title:"Learn javascript",author:"John doe"}
var book2 = {};

in this example all the functions,variables and objects are declared and polluting the global scope of your application . You can refactor this type of code by creating a single global object for your application , called for example Student and change all functions and variables to become properties of your global object

//Declare the global object
var STUDENT = {}
//constructors
STUDENT.Read = function(){};
STUDENT.SPEAK = function(){};

//a varibale
STUDENT.topic_to_learn = "javascript"

//object container 
STUDENT.books = {}

//nested objects 
STUDENT.books.book1 = {};
STUDENT.books.book1.data = {title:"Learn javascript",author:"John doe"}
//add second book
STUDENT.books.book2 = {};

This pattern is good way to namespace your code and avoid naming collisions not only in your own code but collisions between your code and third-party code on the same page .

Drawbacks of Namespace

More to type , prefixing every variable and function adds up in the total amount of code that needs to be downloaded
Only one global instance as a result any part of the code can modify the global instance and the rest of the functionality gets the updated state
Long nested names = slower property resolution lookups

Exploring several types of javascript functions

George Hadjisavva — Thu, 11 Nov 2021 20:16:57 +0000

Returning Functions

In Javascript functions are objects , so they can be used as return values . Therefore functions doesn't need to return some sort of data or array as result of its execution. A function can also return more specialised function or it can generate another function on demand , depending of inputs .

Here's a simple example ,functions does some work and then returns another function , which can also be executed

var login = function () {
    console.log("Hello");
    return function () {
        console.log("World");
    }
}

//Using login function
var hello = login(); //log hello
hello(); // log world

lets see another example ..

var setup = function () {
    var count = 0 ;
    return function() {
        return (count +=1);
    };
};

//usage
var next = setup();
next(); //returns 1
next(); //returns 2
next(); //returns 3

in this example setup wraps the returned function , it creates a closure and you can use this closure to store some private data which is accessible by the returned function only in the scope.

Self-Defining Functions (Lazy function)

Functions can be defined dynamically and can be assigned into variables. You can override the old function with the new one if you create a new function and assign it to the same variable that already holds another function.In this case function overwrites and redefines itself with a new implementation.
To simplify this lets see a simple example

var helpMe = function () {
    alert("help me")
    helpMe = function() {
        alert("Please , Help me")      
    };
};

//Using the self-defining function
helpMe(); // help me
helpMe(); // Please, Help me

Self-defining functions pattern is very useful when your function has some initial preparatory work and it is require to do it only once .
Using this pattern can improve the performance and efficiency of your application .

Immediate Functions(Self-invoking or Self-executing)

The immediate function pattern is syntax that enables you to execute functions as soon as it is defined .Heres an example :

(function () {
    alert("Help");
}())

This pattern is just a function expression (either named or anonymous) which is executed immediately after its creation . The term immediate function term is not defined in the ECMAScript Standard.

Steps for defining immediate function

You define a function using a function expression
You add a set of parentheses at the end, which causes the function to be executed immediately
You wrap the whole function block in parentheses (only if you don't assign the function to a variable

Think the scenario when your code has to perform some setup tasks when the page initially loads ex : creating objects . This needs to be done only once , so creating reusable named function is unnecessary . Thats why you need immediate function , to wrap all the code in its local scope and not leak any variables to the global scope

Passing parameters to immediate function

You have the ability to pass arguments to immediate functions

//Prints : 
//Hello Joe , today is Nov 9 2022 23:26:34 GMT-0800

(function (name,day){
    console.log("Hello" + name + " Today is " + day )
},("Joe",new Date()));

usually the global object(this) is passed as an argument to the immediate function so its accessible inside of the function without having to use window

Note: Avoid passing to many parameters to an immediate function because it could make the function unreadable and difficult to understand ..

Returned Values from immediate Functions

Immediate function can return values and these returns values can be assigned to variables

var result = (function() {
    return 5+5;
}());

You can achieve the same results by omitting the parentheses that wrap the function , because they are not required when you assign the return value to a variable

var result = function() {
    return 5+5;
}();

Immediate functions can also be used when you define objects . A good example to use Immediate function to instantiate object is , lets say you need to define a property that will never change during the life circle of object but before you define it needs to perform a bit of work and the returned value will be the value of property .

Benefits of the Immediate functions
This pattern helps you wrap an amount of work you want to do without leaving any global variables behind. All the defining variables will be local to the self-invoking functions without worry about the global space.
The pattern also enables you to wrap individual features into self-contained modules .