<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jay Ehsaniara</title>
    <description>The latest articles on DEV Community by Jay Ehsaniara (@ehsaniara).</description>
    <link>https://dev.to/ehsaniara</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F338110%2F653e67da-5422-4b60-aa28-b18615cfff4a.png</url>
      <title>DEV Community: Jay Ehsaniara</title>
      <link>https://dev.to/ehsaniara</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ehsaniara"/>
    <language>en</language>
    <item>
      <title>Micro-Containers: Lightweight, Secure Job Execution with Joblet</title>
      <dc:creator>Jay Ehsaniara</dc:creator>
      <pubDate>Wed, 02 Jul 2025 20:57:38 +0000</pubDate>
      <link>https://dev.to/ehsaniara/how-to-use-worker-secure-job-execution-made-simple-559h</link>
      <guid>https://dev.to/ehsaniara/how-to-use-worker-secure-job-execution-made-simple-559h</guid>
      <description>&lt;h1&gt;
  
  
  🧱 Micro-Containers: Lightweight, Secure Job Execution with Joblet
&lt;/h1&gt;

&lt;p&gt;In a world increasingly driven by containerization and distributed workloads, developers and platform engineers face a&lt;br&gt;
growing need for &lt;strong&gt;fast, secure, and minimal overhead execution environments&lt;/strong&gt;. Enter the concept of the *&lt;br&gt;
&lt;em&gt;Micro-Container&lt;/em&gt;* — a pragmatic middle ground between bare processes and full-featured containers like Docker or&lt;br&gt;
Kubernetes pods.&lt;/p&gt;

&lt;p&gt;This article introduces the concept of micro-containers and demonstrates&lt;br&gt;
how &lt;a href="https://github.com/ehsaniara/joblet" rel="noopener noreferrer"&gt;Joblet&lt;/a&gt; brings this idea to life.&lt;/p&gt;


&lt;h2&gt;
  
  
  🚀 What Is a Micro-Container?
&lt;/h2&gt;

&lt;p&gt;A &lt;strong&gt;micro-container&lt;/strong&gt; is an execution unit that provides &lt;strong&gt;process-level isolation and resource control&lt;/strong&gt; without the&lt;br&gt;
heavyweight machinery of traditional containers.&lt;/p&gt;
&lt;h3&gt;
  
  
  Characteristics of a Micro-Container:
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Namespace Isolation&lt;/strong&gt;: Processes are sandboxed using Linux PID, mount, UTS, and network namespaces.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resource Limits&lt;/strong&gt;: CPU, memory, I/O, and PID caps enforced via Linux cgroups v2.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Filesystem Segregation&lt;/strong&gt;: Executed inside a chroot environment or limited mount namespace.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Startup in Milliseconds&lt;/strong&gt;: No image pull, no daemon — just fork and isolate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Security-First&lt;/strong&gt;: TLS/mTLS, user-based access control, no root access required.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This lightweight model is ideal for use cases like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ephemeral job runners&lt;/li&gt;
&lt;li&gt;Secure execution of user-submitted code&lt;/li&gt;
&lt;li&gt;CI/CD pipeline steps&lt;/li&gt;
&lt;li&gt;Scripted automation in controlled environments&lt;/li&gt;
&lt;/ul&gt;


&lt;h2&gt;
  
  
  🛠️ Enter Joblet: A Micro-Container Engine
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/ehsaniara/joblet" rel="noopener noreferrer"&gt;&lt;strong&gt;Joblet&lt;/strong&gt;&lt;/a&gt; is an open-source project that implements the micro-container model&lt;br&gt;
for real-world Linux workloads. It abstracts process isolation and resource management behind a &lt;strong&gt;simple CLI&lt;/strong&gt; and *&lt;br&gt;
&lt;em&gt;gRPC API&lt;/em&gt;*, enabling secure, ephemeral job execution with minimal configuration.&lt;/p&gt;
&lt;h3&gt;
  
  
  Core Features of Joblet:
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;✅ &lt;strong&gt;Linux-native isolation&lt;/strong&gt;: Uses namespaces, cgroups, and chroot (no Docker/OCI).&lt;/li&gt;
&lt;li&gt;✅ &lt;strong&gt;gRPC API&lt;/strong&gt;: Programmatically launch, monitor, and stream jobs.&lt;/li&gt;
&lt;li&gt;✅ &lt;strong&gt;Real-time logging&lt;/strong&gt;: Get stdout/stderr via SSE or logs endpoint.&lt;/li&gt;
&lt;li&gt;✅ &lt;strong&gt;Resource quotas&lt;/strong&gt;: Limit CPU shares, memory, and block I/O per job.&lt;/li&gt;
&lt;li&gt;✅ &lt;strong&gt;Security&lt;/strong&gt;: Enforced TLS + role-based access control (Admin, Viewer).&lt;/li&gt;
&lt;/ul&gt;


&lt;h2&gt;
  
  
  🧬 Under the Hood
&lt;/h2&gt;

&lt;p&gt;Joblet orchestrates Linux primitives in a tightly integrated flow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Namespace Isolation&lt;/strong&gt;: Each job runs in its own PID, mount, and optionally net namespace.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Filesystem Sandboxing&lt;/strong&gt;: With &lt;code&gt;chroot&lt;/code&gt; and custom mount points, jobs are restricted from accessing host files.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;cgroups v2 Enforcement&lt;/strong&gt;: Jobs are assigned to dynamically created cgroups with strict resource boundaries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Process Lifecycle Management&lt;/strong&gt;: All subprocesses are cleaned up upon exit or timeout.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Logging &amp;amp; Monitoring&lt;/strong&gt;: Logs are streamed via Server-Sent Events (SSE), while job metadata (duration, status,
resource usage) is tracked internally.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Here’s what launching a secure job might look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;joblet run &lt;span class="nt"&gt;--cpu&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0.5 &lt;span class="nt"&gt;--mem&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;256M &lt;span class="nt"&gt;--chroot&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/sandbox ./my-script.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or via API:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight protobuf"&gt;&lt;code&gt;&lt;span class="k"&gt;rpc&lt;/span&gt; &lt;span class="n"&gt;SubmitJob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;JobRequest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;returns&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;JobResponse&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  🔬 Micro-Containers vs Containers
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Feature&lt;/th&gt;
&lt;th&gt;Micro-Container (Joblet)&lt;/th&gt;
&lt;th&gt;Docker Container&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Startup Time&lt;/td&gt;
&lt;td&gt;~20–100ms&lt;/td&gt;
&lt;td&gt;400–2000ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Daemon Required&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Images Needed&lt;/td&gt;
&lt;td&gt;❌ (just binaries/scripts)&lt;/td&gt;
&lt;td&gt;✅ (Dockerfile etc.)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Isolation Strength&lt;/td&gt;
&lt;td&gt;High (namespace + cgroup)&lt;/td&gt;
&lt;td&gt;Very High&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Filesystem&lt;/td&gt;
&lt;td&gt;Host + chroot&lt;/td&gt;
&lt;td&gt;Layered FS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Use Case&lt;/td&gt;
&lt;td&gt;CI steps, job runners&lt;/td&gt;
&lt;td&gt;Services, microservices&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  🔐 When to Use Micro-Containers
&lt;/h2&gt;

&lt;p&gt;Joblet’s model shines in situations where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You &lt;strong&gt;don’t want the overhead of Docker or Kubernetes&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;You need &lt;strong&gt;fine-grained CPU/memory control&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;You execute &lt;strong&gt;untrusted or dynamic code&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;You want &lt;strong&gt;fast, low-latency job startup&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;You’re building &lt;strong&gt;multi-tenant pipelines or judges&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  🌍 Real-World Use Cases
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Online Code Judges&lt;/strong&gt;: Securely run user code submissions in isolated sandboxes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internal CI/CD Steps&lt;/strong&gt;: Replace Docker steps with instant-on, native jobs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data Pipelines&lt;/strong&gt;: Execute preprocessing and validation stages as atomic jobs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Self-Hosted Serverless Platform&lt;/strong&gt;: Accept jobs over gRPC and execute them with guarantees.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  🧪 Conclusion
&lt;/h2&gt;

&lt;p&gt;The “micro-container” model strikes a powerful balance between raw Linux process execution and full-blown containers. It&lt;br&gt;
gives platform engineers the primitives to isolate, monitor, and control processes &lt;strong&gt;without needing Docker, Kubernetes,&lt;br&gt;
or VMs&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Joblet&lt;/strong&gt; brings this model to life with a clean, efficient, and secure implementation — a single binary that gives you&lt;br&gt;
safe multi-tenant job execution out of the box.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Ready to try it?&lt;br&gt;
👉 &lt;a href="https://github.com/ehsaniara/joblet" rel="noopener noreferrer"&gt;GitHub – Joblet&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;p&gt;Let me know if you'd like this converted into a Markdown article for GitHub Pages or Dev.to, or if you'd like a logo or&lt;br&gt;
diagram to accompany it.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>go</category>
      <category>systems</category>
      <category>programming</category>
    </item>
    <item>
      <title>gRPC between Web and Server.</title>
      <dc:creator>Jay Ehsaniara</dc:creator>
      <pubDate>Fri, 30 Aug 2024 14:32:10 +0000</pubDate>
      <link>https://dev.to/ehsaniara/grpc-between-web-and-server-7k6</link>
      <guid>https://dev.to/ehsaniara/grpc-between-web-and-server-7k6</guid>
      <description>&lt;p&gt;This project demonstrates how to set up a simple gRPC communication between a web client and server with an Envoy proxy.&lt;/p&gt;

&lt;p&gt;I’m using the gRPC-Web library for this purpose. gRPC-Web is a JavaScript client library that allows web applications to interact with gRPC services. Since browsers don't support HTTP/2 or the binary protocol used by standard gRPC, gRPC-Web provides a way to bridge the gap by using HTTP/1.1 or HTTP/2 and encoding gRPC messages in a way that browsers can handle. Here's how gRPC-Web works:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The client sends requests to the server using gRPC-Web, which typically communicates via HTTP/1.1 or HTTP/2. Metadata (like headers) can be attached to the request, such as for authentication (e.g., JWT tokens).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The request is encoded in a gRPC-Web format, typically using base64 encoding for the binary gRPC payload. The client sends this request over HTTP/1.1 or HTTP/2.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Envoy (or another reverse proxy like Nginx) acts as an intermediary between the gRPC-Web client and the gRPC server. Envoy receives the gRPC-Web request, decodes the gRPC-Web payload, and forwards it as a standard gRPC request to the gRPC server using HTTP/2.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The gRPC server processes the request as if it were a native gRPC request, using HTTP/2 for communication.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;The gRPC server processes the incoming gRPC request, performs the necessary business logic, and generates a response (in this example it’s the Go written application). The response is encoded in the standard gRPC format and sent back to Envoy.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Envoy receives the gRPC response, encodes it in the gRPC-Web format (typically using base64), and sends it back to the gRPC-Web client over HTTP/1.1 or HTTP/2. Any metadata included in the gRPC response, such as status codes, is translated appropriately.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The gRPC-Web client decodes the response and converts it into a usable format within the web application. The web application processes the response, updating the UI or handling errors as needed.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fl4k7v50p9attuezozi1l.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fl4k7v50p9attuezozi1l.png" alt="Image description" width="731" height="221"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;NOTE: client-side and bi-directional streaming is not currently supported (see streaming roadmap)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Advantages of gRPC-Web
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Browser Compatibility: Allows modern web applications to interact with gRPC services without needing native support for HTTP/2 and binary protocols.&lt;/li&gt;
&lt;li&gt;Efficiency: Leverages the performance and efficiency of gRPC while adapting it for the web.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here is a GitHub project for it:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/ehsaniara/gRPC-web-example" rel="noopener noreferrer"&gt;https://github.com/ehsaniara/gRPC-web-example&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  proto file
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;syntax = "proto3";

package helloworld;

option go_package = "./proto"; // Add this line

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  server side (go)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;package main

import (
   "context"
   "google.golang.org/grpc/reflection"
   "log"
   "net"

   pb "github.com/ehsaniara/gRPC-web-example/proto"
   "google.golang.org/grpc"
)

type server struct {
 pb.UnimplementedGreeterServer
}

func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
 return &amp;amp;pb.HelloReply{Message: "Hello " + in.Name}, nil
}

func main() {
 lis, err := net.Listen("tcp", ":50051")
 if err != nil {
  log.Fatalf("failed to listen: %v", err)
 }
 s := grpc.NewServer()
 pb.RegisterGreeterServer(s, &amp;amp;server{})

 // Register reflection service on gRPC server.
 reflection.Register(s)
 if err := s.Serve(lis); err != nil {
  log.Fatalf("failed to serve: %v", err)
 }

 log.Println("Server is running on port 50051")
 if err := s.Serve(lis); err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Envoy config
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;...
http_filters:
  - name: envoy.filters.http.grpc_web
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
  - name: envoy.filters.http.cors
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
  - name: envoy.filters.http.router
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  JS Web Client (webpack)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;// Import the generated gRPC-Web client stubs and message classes
import {GreeterClient} from './generated/helloworld_grpc_web_pb';
import {HelloRequest} from './generated/helloworld_pb';

// Create an instance of the Greeter client
const client = new GreeterClient('http://localhost:8080');

// Function to send a greeting request
function sayHello(name) {
    // Create a new request
    const request = new HelloRequest();
    request.setName(name);

    // Call the sayHello method on the Greeter client
    client.sayHello(request, {}, (err, response) =&amp;gt; {
        if (err) {
            console.error('Error:', err.message);
            document.getElementById('output').textContent = 'Error: ' + err.message;
        } else {
            console.log('Greeting:', response.getMessage());
            document.getElementById('output').textContent = 'Greeting: ' + response.getMessage();
        }
    });
}

// Example usage: sending a request when the page loads
document.addEventListener('DOMContentLoaded', () =&amp;gt; {
    const name = 'World';
    sayHello(name);
});
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here is a GitHub project related to this project&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/ehsaniara/gRPC-web-example" rel="noopener noreferrer"&gt;https://github.com/ehsaniara/gRPC-web-example&lt;/a&gt;&lt;/p&gt;

</description>
      <category>grpc</category>
      <category>web</category>
      <category>go</category>
      <category>browser</category>
    </item>
    <item>
      <title>Go Interval job timer, with centralized Lock for Distributed Systems</title>
      <dc:creator>Jay Ehsaniara</dc:creator>
      <pubDate>Sun, 01 Aug 2021 02:32:40 +0000</pubDate>
      <link>https://dev.to/ehsaniara/go-task-scheduler-with-distributed-lock-4am1</link>
      <guid>https://dev.to/ehsaniara/go-task-scheduler-with-distributed-lock-4am1</guid>
      <description>&lt;h1&gt;
  
  
  goInterLock
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;known as: ⏰ Interval (Cron / Job / Task / Scheduler) ⏱️&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;GoInterval&lt;/strong&gt; job timer, with centralize &lt;strong&gt;Lock&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;goInterLock&lt;/code&gt; is schedule tasks (cron-job) with a centralized locking mechanism for Go (AKA Golang). In the distributed systems, the lock prevents the tasks to been executed in every instant that has the scheduler.&lt;/p&gt;

&lt;p&gt;For example, if your application has a scheduled where it's calling some external API or doing some expensive DataBase querying every 10 minutes, the lock prevents the process to be run in every instance of that application, and you ended up running that task multiple times every 10 minutes. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ideal use cases:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Kubernetes&lt;/li&gt;
&lt;li&gt;Docker Swarm&lt;/li&gt;
&lt;li&gt;AWS ECS Fargate
Quick Start
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go get github.com/ehsaniara/gointerlock
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Supported Lock&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Redis&lt;/li&gt;
&lt;li&gt;AWS DynamoDB&lt;/li&gt;
&lt;li&gt;Postgres DB (coming soon)&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  Local Scheduler (Single App)
&lt;/h1&gt;

&lt;p&gt;(Interval every 2 seconds)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gointerlock&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GoInterval&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Interval&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Arg&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;      &lt;span class="n"&gt;myJob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Error: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Examples
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://github.com/ehsaniara/gointerlock/blob/main/example/basicLocal/main.go" rel="noopener noreferrer"&gt;&lt;strong&gt;Basic Local Task:&lt;/strong&gt;&lt;/a&gt; Simple Task Interval (Single App).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/ehsaniara/gointerlock/blob/main/example/applicationCache/main.go" rel="noopener noreferrer"&gt;&lt;strong&gt;Application Cache:&lt;/strong&gt;&lt;/a&gt; An example of periodically cached value update on http server.&lt;/p&gt;




&lt;h1&gt;
  
  
  Distributed Mode (Scaled Up)
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Redis
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Existing Redis Connection
&lt;/h3&gt;

&lt;p&gt;you should already configure your Redis connection and pass it into the &lt;code&gt;GoInterLock&lt;/code&gt;. Also make sure you are giving the&lt;br&gt;
unique name per job&lt;/p&gt;

&lt;p&gt;Step 1: configure redis connection &lt;code&gt;redisConnection.Rdb&lt;/code&gt; from the existing application and pass it into the Job. for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;redisConnector&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Options&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Addr&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;     &lt;span class="s"&gt;"localhost:6379"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Password&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"myRedisPassword"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
    &lt;span class="n"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;       &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;               
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Step 2: Pass the redis connection into the &lt;code&gt;GoInterval&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gointerlock&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GoInterval&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Interval&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Arg&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;      &lt;span class="n"&gt;myJob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;     &lt;span class="s"&gt;"MyTestJob"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;RedisConnector&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;redisConnector&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;jobTicker&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Error: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;in both examples &lt;code&gt;myJob&lt;/code&gt; is your function, for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;myJob&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="s"&gt;" - called"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;Note: currently &lt;code&gt;GoInterLock&lt;/code&gt; does not support any argument for the job function&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Built in Redis Connector
&lt;/h3&gt;

&lt;p&gt;another way is to use an existing redis connection:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gointerlock&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GoInterval&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;          &lt;span class="s"&gt;"MyTestJob"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Interval&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;      &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Arg&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;           &lt;span class="n"&gt;myJob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;RedisHost&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;     &lt;span class="s"&gt;"localhost:6379"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;RedisPassword&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"myRedisPassword"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c"&gt;//if no pass leave it as ""&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Error: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h5&gt;
  
  
  GoInterLock is using &lt;a href="https://github.com/go-redis/redis" rel="noopener noreferrer"&gt;go-redis&lt;/a&gt; for Redis Connection.
&lt;/h5&gt;

&lt;h3&gt;
  
  
  Examples
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://github.com/ehsaniara/gointerlock/blob/main/example/redis/basic/main.go" rel="noopener noreferrer"&gt;&lt;strong&gt;Basic Distributed Task:&lt;/strong&gt;&lt;/a&gt; Simple Task Interval with Redis Lock.&lt;/p&gt;




&lt;h2&gt;
  
  
  AWS DynamoDb
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Basic Config (Local Environment)
&lt;/h3&gt;

&lt;p&gt;This ia sample of local DynamoDb (Docker) for your local test.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gointerlock&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GoInterval&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                       &lt;span class="s"&gt;"MyTestJob"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Interval&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                   &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Arg&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                        &lt;span class="n"&gt;myJob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;LockVendor&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                 &lt;span class="n"&gt;gointerlock&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AwsDynamoDbLock&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;AwsDynamoDbRegion&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;          &lt;span class="s"&gt;"us-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;AwsDynamoDbEndpoint&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;        &lt;span class="s"&gt;"http://127.0.0.1:8000"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;AwsDynamoDbSecretAccessKey&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"dummy"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;AwsDynamoDbAccessKeyID&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;     &lt;span class="s"&gt;"dummy"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cnx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Error: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;task:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;myJob&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="s"&gt;" - called"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note: you can get the docker-compose file from &lt;a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html" rel="noopener noreferrer"&gt;AWS DynamoDB Docker compose&lt;/a&gt; or you can get it from: &lt;a href="https://github.com/ehsaniara/gointerlock/blob/main/example/awsDynamoDb/docker-compose.yml" rel="noopener noreferrer"&gt;docker-compose.yml&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Using AWS Profile
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;goInterLock&lt;/code&gt; will get credentials from the AWS profile&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gointerlock&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GoInterval&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                       &lt;span class="s"&gt;"MyTestJob"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Interval&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                   &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Arg&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                        &lt;span class="n"&gt;myJob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;LockVendor&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;                 &lt;span class="n"&gt;gointerlock&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AwsDynamoDbLock&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cnx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Error: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Examples
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://github.com/ehsaniara/gointerlock/blob/main/example/awsDynamoDb/main.go" rel="noopener noreferrer"&gt;&lt;strong&gt;Basic Distributed Task:&lt;/strong&gt;&lt;/a&gt; Simple Task Interval with DynamoDb Lock.&lt;/p&gt;

&lt;h1&gt;
  
  
  🍀 Just the centralized lock for distributed system (No Timer/Scheduler)
&lt;/h1&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"context"&lt;/span&gt;
    &lt;span class="s"&gt;"fmt"&lt;/span&gt;
    &lt;span class="s"&gt;"time"&lt;/span&gt;

    &lt;span class="s"&gt;"github.com/go-redis/redis/v8"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;DistributedLock&lt;/span&gt; &lt;span class="k"&gt;interface&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockName&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;maxLockingDuration&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;
    &lt;span class="n"&gt;UnLock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockName&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;NewDistributedLock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rdb&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;DistributedLock&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;distributedLock&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;rdb&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rdb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;distributedLock&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;rdb&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Client&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;// Lock return TRUE when successfully locked, return FALSE if it's already been locked by others&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="n"&gt;distributedLock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockName&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;maxLockingDuration&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"lock_%s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockName&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c"&gt;//check if it's already locked&lt;/span&gt;
    &lt;span class="n"&gt;iter&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rdb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Scan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Iterator&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;iter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c"&gt;//exit if lock exist&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;false&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="c"&gt;//then lock it then&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rdb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="kt"&gt;byte&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"true"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;maxLockingDuration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="n"&gt;distributedLock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;UnLock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockName&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"lock_%s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockName&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c"&gt;//remove the lock&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rdb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Del&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



</description>
      <category>go</category>
      <category>distributedsystems</category>
      <category>redis</category>
      <category>dynamodb</category>
    </item>
    <item>
      <title>ElasticSearch Parallel Pagination by Kafka</title>
      <dc:creator>Jay Ehsaniara</dc:creator>
      <pubDate>Sat, 09 Jan 2021 20:09:38 +0000</pubDate>
      <link>https://dev.to/ehsaniara/elasticsearch-parallel-pagination-by-kafka-304a</link>
      <guid>https://dev.to/ehsaniara/elasticsearch-parallel-pagination-by-kafka-304a</guid>
      <description>&lt;p&gt;By Using Kafka we can easily horizontally scale our application to do asynchronous pagination in ElasticSearch.&lt;/p&gt;

&lt;p&gt;Let’s say you have an ElasticSearch Index of 1,000,000 documents, and you need to run an operation on those documents. We already know how expensive the deep-paging in ElasticSearch is, especially index.max_result_window and doing Search ‘from:’.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;GET /_search
&lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;"from"&lt;/span&gt;: 5,
  &lt;span class="s2"&gt;"size"&lt;/span&gt;: 20,
  &lt;span class="s2"&gt;"query"&lt;/span&gt;: &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"match"&lt;/span&gt;: &lt;span class="o"&gt;{&lt;/span&gt;
      &lt;span class="s2"&gt;"user.id"&lt;/span&gt;: &lt;span class="s2"&gt;"jay"&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One way of overcoming the problem is to use search_after. In this case, your process is becoming synchronous, which means you cannot call the second chunk of data without having the results from the first call, for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;GET /_search
&lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;"size"&lt;/span&gt;: 10000,
  &lt;span class="s2"&gt;"query"&lt;/span&gt;: &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"match"&lt;/span&gt; : &lt;span class="o"&gt;{&lt;/span&gt;
      &lt;span class="s2"&gt;"user.id"&lt;/span&gt; : &lt;span class="s2"&gt;"jay"&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;}&lt;/span&gt;,
  &lt;span class="s2"&gt;"sort"&lt;/span&gt;: &lt;span class="o"&gt;[&lt;/span&gt; 
    &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"@timestamp"&lt;/span&gt;: &lt;span class="s2"&gt;"asc"&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;results:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;"took"&lt;/span&gt; : ...,
  &lt;span class="s2"&gt;"timed_out"&lt;/span&gt; : &lt;span class="nb"&gt;false&lt;/span&gt;,
  &lt;span class="s2"&gt;"_shards"&lt;/span&gt; : ...,
  &lt;span class="s2"&gt;"hits"&lt;/span&gt; : &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"total"&lt;/span&gt; : ...,
    &lt;span class="s2"&gt;"max_score"&lt;/span&gt; : null,
    &lt;span class="s2"&gt;"hits"&lt;/span&gt; : &lt;span class="o"&gt;[&lt;/span&gt;
      ...
      &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"_source"&lt;/span&gt; : ...,
        &lt;span class="s2"&gt;"sort"&lt;/span&gt; : &lt;span class="o"&gt;[&lt;/span&gt;                                
          4098435132000
        &lt;span class="o"&gt;]&lt;/span&gt;
      &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;]&lt;/span&gt;
  &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As you can see, the results give you a sort value (4098435132000) where you can use in your second call-in “search_after” to get the next chunk as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;GET /_search
&lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;"size"&lt;/span&gt;: 10000,
  &lt;span class="s2"&gt;"query"&lt;/span&gt;: &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"match"&lt;/span&gt; : &lt;span class="o"&gt;{&lt;/span&gt;
      &lt;span class="s2"&gt;"user.id"&lt;/span&gt; : &lt;span class="s2"&gt;"jay"&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;}&lt;/span&gt;,
  &lt;span class="s2"&gt;"sort"&lt;/span&gt;: &lt;span class="o"&gt;[&lt;/span&gt;
    &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"@timestamp"&lt;/span&gt;: &lt;span class="s2"&gt;"asc"&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;]&lt;/span&gt;,
  &lt;span class="s2"&gt;"search_after"&lt;/span&gt;: &lt;span class="o"&gt;[&lt;/span&gt;                                
    4098435132000
  &lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;then get the next sort value and use it in your next call.&lt;/p&gt;

&lt;p&gt;This process is sequentially asynchronous, which means to get through 1,000,000 documents you need to call ES 1,000,000/10,000=100 times one after the other.&lt;/p&gt;

&lt;h3&gt;
  
  
  What can go wrong?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;let’s say your application dies or runs out of memory during these calls&lt;/li&gt;
&lt;li&gt;how to know where to start after it failed&lt;/li&gt;
&lt;li&gt;maybe this process is very slow for your application use-case&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Solution By Using “Slice”
&lt;/h2&gt;

&lt;p&gt;One call to ES to get the count of the documents (which in this case is 1M), then create a Kafka producer to put the sequences (0–100) into your Kafka topic and define your partition size as the number of your consumer app. Let’s say you have 10 consumer applications.&lt;/p&gt;

&lt;p&gt;Create a Kafka consumer app, which is calling Elastic Search as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;GET /_search?scroll&lt;span class="o"&gt;=&lt;/span&gt;1m
&lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;"slice"&lt;/span&gt;: &lt;span class="o"&gt;{&lt;/span&gt;
     &lt;span class="s2"&gt;"id"&lt;/span&gt;:0,
     &lt;span class="s2"&gt;"max"&lt;/span&gt;:100  
  &lt;span class="o"&gt;}&lt;/span&gt;,
  &lt;span class="s2"&gt;"size"&lt;/span&gt;: 10000,
  &lt;span class="s2"&gt;"sort"&lt;/span&gt;: &lt;span class="o"&gt;[&lt;/span&gt;
    &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"@timestamp"&lt;/span&gt;: &lt;span class="s2"&gt;"asc"&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The consumer gets the sequence number as a parameter and replaces it with the id. As long as all consumers have the same consumer “group.id” they can process these calls in parallel and have a resilient framework.&lt;/p&gt;

&lt;h2&gt;
  
  
  Implementation with Java/Spring Cloud
&lt;/h2&gt;

&lt;p&gt;I happened to use java with spring cloud for my project.&lt;/p&gt;

&lt;p&gt;Let’s run your Elastic Search and Kafka in docker-compose by running the following command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Elastic Search&lt;/span&gt;
docker-compose &lt;span class="nt"&gt;-f&lt;/span&gt; docker-compose-es.yml up &lt;span class="nt"&gt;-d&lt;/span&gt;
&lt;span class="c"&gt;# Kafka&lt;/span&gt;
docker-compose &lt;span class="nt"&gt;-f&lt;/span&gt; docker-compose-kafka.yml up &lt;span class="nt"&gt;-d&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;you can find the docker-compose-es.yml and docker-compose-kafka.yml in my Github account.&lt;/p&gt;

&lt;p&gt;The producer first calls ES to get the count of the documents in the index, then divides the count by 500, which will be the number of documents you expect to be retrieved in every ES call.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;paginationProcess&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;debug&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"paginationProcess called"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="c1"&gt;//call to ES and get the total count&lt;/span&gt;
    &lt;span class="nc"&gt;Response&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;restHighLevelClient&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getLowLevelClient&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;performRequest&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"GET"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"%s/_count"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="no"&gt;INDEX_NAME&lt;/span&gt;&lt;span class="o"&gt;)));&lt;/span&gt;

    &lt;span class="nc"&gt;ResponseCountDto&lt;/span&gt; &lt;span class="n"&gt;responseCountDto&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;objectMapper&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;readValue&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;EntityUtils&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toString&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getEntity&lt;/span&gt;&lt;span class="o"&gt;()),&lt;/span&gt; &lt;span class="nc"&gt;ResponseCountDto&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;debug&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"responseCountDto: {}"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;responseCountDto&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getCount&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;

    &lt;span class="c1"&gt;//let say you want to have a page size of 500 then count / 500&lt;/span&gt;

    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;max&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;responseCountDto&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getCount&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;debug&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"count: {} , max: {}"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;responseCountDto&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getCount&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

    &lt;span class="c1"&gt;//producer&lt;/span&gt;
    &lt;span class="nc"&gt;IntStream&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;range&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;forEach&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;paginationBinder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;paginationOut&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="c1"&gt;//&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;send&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;MessageBuilder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;withPayload&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="c1"&gt;//&lt;/span&gt;
    &lt;span class="nc"&gt;PaginationDto&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;builder&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="c1"&gt;//&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;//slice id&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;max&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;// let say i want to have page size of 500 then: count / 500&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;())&lt;/span&gt;&lt;span class="c1"&gt;//&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setHeader&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;MessageHeaders&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;CONTENT_TYPE&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;MimeTypeUtils&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;APPLICATION_JSON&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;()));&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the Consumer receives the slice number as a parameter. In case of any error in the consumer method, we retry it 5 times, and then it puts the message into to DLQ method.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt; &lt;span class="nd"&gt;@StreamListener&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;PaginationBinder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;PAGINATION_IN&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;paginationProcess&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nd"&gt;@Payload&lt;/span&gt; &lt;span class="nc"&gt;PaginationDto&lt;/span&gt; &lt;span class="n"&gt;paginationDto&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// Call ES&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;debug&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"paginationProcess: {}"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;paginationDto&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="nc"&gt;Request&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"GET"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"%s/_search?scroll=1m"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="no"&gt;INDEX_NAME&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
            &lt;span class="c1"&gt;//sorted by localDateTime and slice by id and max as parameters&lt;/span&gt;
            &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setJsonEntity&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"{\"slice\":{\"id\":%s,\"max\":%s},\"size\":10000,\"sort\":[{\"localDateTime\":\"asc\"}]}"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;paginationDto&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getId&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;paginationDto&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getMax&lt;/span&gt;&lt;span class="o"&gt;()));&lt;/span&gt;
            &lt;span class="nc"&gt;Response&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;restHighLevelClient&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getLowLevelClient&lt;/span&gt;&lt;span class="o"&gt;().&lt;/span&gt;&lt;span class="na"&gt;performRequest&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="c1"&gt;//do something with the response ...&lt;/span&gt;
            &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;debug&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"response: {}"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;IOException&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;printStackTrace&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note: Assume no new documents are inserting into the result set or at least within that period. Otherwise, we lose the consistency of the result during the pagination.&lt;/p&gt;

&lt;p&gt;application.yml file&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spring&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;application&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;es-pagination&lt;/span&gt;

  &lt;span class="na"&gt;cloud.stream&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;bindings&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;

      &lt;span class="na"&gt;pagination-out&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;destination&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pagination&lt;/span&gt;
        &lt;span class="na"&gt;producer&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;partition-count&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt;
      &lt;span class="na"&gt;pagination-in&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;destination&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pagination&lt;/span&gt;
        &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${spring.application.name}.pagination-group&lt;/span&gt;
        &lt;span class="na"&gt;consumer&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;maxAttempts&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
      &lt;span class="na"&gt;pagination-in-dlq&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;destination&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;paginationDLQ&lt;/span&gt;
        &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${spring.application.name}.pagination-group&lt;/span&gt;

    &lt;span class="na"&gt;kafka&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;streams&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;bindings&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;pagination-in&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;consumer&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;enableDlq&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
              &lt;span class="na"&gt;dlqName&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;paginationDLQ&lt;/span&gt;
              &lt;span class="na"&gt;autoCommitOnError&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
              &lt;span class="na"&gt;autoCommitOffset&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
        &lt;span class="na"&gt;binder&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;autoAddPartitions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;min-partition-count&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt;
          &lt;span class="na"&gt;configuration&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;commit.interval.ms&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt;
            &lt;span class="na"&gt;default.key.serde&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org.apache.kafka.common.serialization.Serdes$StringSerde&lt;/span&gt;
            &lt;span class="na"&gt;default.value.serde&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org.apache.kafka.common.serialization.Serdes$StringSerde&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Avoid toasting the memory
&lt;/h3&gt;

&lt;p&gt;If the number of slices is bigger than the number of shards, the slice filter will become very slow on the first calls. It has a complexity of O(N) and a memory cost which equals N bits per slice where N is the total number of documents in the shard. After a few calls, the filter should be cached and subsequent calls should be faster, but you should limit the number of sliced queries you perform in parallel to avoid the memory explosion.&lt;/p&gt;

&lt;p&gt;You can find the project in my Github account:&lt;br&gt;
&lt;a href="https://github.com/ehsaniara/es-kafka-pagination" rel="noopener noreferrer"&gt;https://github.com/ehsaniara/es-kafka-pagination&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Refrence:&lt;br&gt;
&lt;a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html" rel="noopener noreferrer"&gt;https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html&lt;/a&gt;&lt;/p&gt;

</description>
      <category>elasticsearch</category>
      <category>kafka</category>
      <category>springcloud</category>
      <category>java</category>
    </item>
    <item>
      <title>Spring Boot 2 with Multiple DataSources</title>
      <dc:creator>Jay Ehsaniara</dc:creator>
      <pubDate>Wed, 19 Feb 2020 08:18:05 +0000</pubDate>
      <link>https://dev.to/ehsaniara/spring-boot-2-with-multiple-datasources-1n9e</link>
      <guid>https://dev.to/ehsaniara/spring-boot-2-with-multiple-datasources-1n9e</guid>
      <description>&lt;h1&gt;
  
  
  Spring Boot 2 with Multiple DataSource
&lt;/h1&gt;

&lt;p&gt;There are times that even having the best DataBase (PostgresSql, Oracle, MySql, .. ) Tuning can not be as help-full as Application Level separating Read and Writes&lt;/p&gt;

&lt;p&gt;Spring Boot 2.2.2 with Multiple DataSource &lt;/p&gt;

&lt;h2&gt;
  
  
  Postgres Setup
&lt;/h2&gt;

&lt;p&gt;For This Demo you need 2 separate Postgres DataBase where one as Master and the other re one as a replica. &lt;/p&gt;

&lt;p&gt;for simplicity just run:&lt;br&gt;
&lt;/p&gt;

&lt;p&gt;&lt;code&gt;docker-compose up --force-recreate&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;docker-compose.yml&lt;/strong&gt; is already in the project which contains 2 PostgresSql in 2 different ports, with &lt;strong&gt;demo&lt;/strong&gt; DataBase&lt;/p&gt;

&lt;h5&gt;
  
  
  Note: you can always uninstall it as: &lt;strong&gt;docker-compose down&lt;/strong&gt; if you needed to.
&lt;/h5&gt;

&lt;h5&gt;
  
  
  you can clone this project from my github on &lt;a href="https://github.com/ehsaniara/spring-boot-multi-data-source" rel="noopener noreferrer"&gt;https://github.com/ehsaniara/spring-boot-multi-data-source&lt;/a&gt;
&lt;/h5&gt;

&lt;h2&gt;
  
  
  Spring Boot Setup
&lt;/h2&gt;

&lt;p&gt;From &lt;a href="https://start.spring.io/" rel="noopener noreferrer"&gt;https://start.spring.io/&lt;/a&gt; select &lt;strong&gt;web&lt;/strong&gt;, &lt;strong&gt;data-jpa&lt;/strong&gt;, &lt;strong&gt;lombok&lt;/strong&gt;, &lt;strong&gt;postgresDriver&lt;/strong&gt; &lt;/p&gt;

&lt;p&gt;Once you Generate and download the zip file, you should have similar POM file as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight xml"&gt;&lt;code&gt;
&lt;span class="nt"&gt;&amp;lt;dependencies&amp;gt;&lt;/span&gt;

   &lt;span class="nt"&gt;&amp;lt;dependency&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;groupId&amp;gt;&lt;/span&gt;org.springframework.boot&lt;span class="nt"&gt;&amp;lt;/groupId&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;artifactId&amp;gt;&lt;/span&gt;spring-boot-starter-web&lt;span class="nt"&gt;&amp;lt;/artifactId&amp;gt;&lt;/span&gt;
   &lt;span class="nt"&gt;&amp;lt;/dependency&amp;gt;&lt;/span&gt;

   &lt;span class="nt"&gt;&amp;lt;dependency&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;groupId&amp;gt;&lt;/span&gt;org.springframework.boot&lt;span class="nt"&gt;&amp;lt;/groupId&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;artifactId&amp;gt;&lt;/span&gt;spring-boot-starter-data-jpa&lt;span class="nt"&gt;&amp;lt;/artifactId&amp;gt;&lt;/span&gt;
   &lt;span class="nt"&gt;&amp;lt;/dependency&amp;gt;&lt;/span&gt;

   &lt;span class="nt"&gt;&amp;lt;dependency&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;groupId&amp;gt;&lt;/span&gt;org.springframework.boot&lt;span class="nt"&gt;&amp;lt;/groupId&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;artifactId&amp;gt;&lt;/span&gt;spring-boot-configuration-processor&lt;span class="nt"&gt;&amp;lt;/artifactId&amp;gt;&lt;/span&gt;
   &lt;span class="nt"&gt;&amp;lt;/dependency&amp;gt;&lt;/span&gt;

   &lt;span class="nt"&gt;&amp;lt;dependency&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;groupId&amp;gt;&lt;/span&gt;org.projectlombok&lt;span class="nt"&gt;&amp;lt;/groupId&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;artifactId&amp;gt;&lt;/span&gt;lombok&lt;span class="nt"&gt;&amp;lt;/artifactId&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;optional&amp;gt;&lt;/span&gt;true&lt;span class="nt"&gt;&amp;lt;/optional&amp;gt;&lt;/span&gt;
   &lt;span class="nt"&gt;&amp;lt;/dependency&amp;gt;&lt;/span&gt;

   &lt;span class="nt"&gt;&amp;lt;dependency&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;groupId&amp;gt;&lt;/span&gt;org.postgresql&lt;span class="nt"&gt;&amp;lt;/groupId&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;artifactId&amp;gt;&lt;/span&gt;postgresql&lt;span class="nt"&gt;&amp;lt;/artifactId&amp;gt;&lt;/span&gt;
      &lt;span class="nt"&gt;&amp;lt;scope&amp;gt;&lt;/span&gt;runtime&lt;span class="nt"&gt;&amp;lt;/scope&amp;gt;&lt;/span&gt;
   &lt;span class="nt"&gt;&amp;lt;/dependency&amp;gt;&lt;/span&gt;


&lt;span class="nt"&gt;&amp;lt;/dependencies&amp;gt;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;for this demo I use HikariDataSource as a default connection pool library by Spring Boot 2.2.2&lt;br&gt;
we need to have 2 separate DataSource and EntityManager one for the Writes(Master/Primary) and one for Reads(Slave/Secondary).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;
&lt;span class="na"&gt;spring&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;datasource-write&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;driver-class-name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org.postgresql.Driver&lt;/span&gt;
    &lt;span class="na"&gt;jdbc-url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;jdbc:postgresql://localhost:5432/demo&lt;/span&gt;
    &lt;span class="na"&gt;username&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;postgres_user_for_db_write'&lt;/span&gt;
    &lt;span class="na"&gt;password&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;you_password'&lt;/span&gt;
    &lt;span class="na"&gt;platform&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;postgresql&lt;/span&gt;
    &lt;span class="na"&gt;hikari&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;idle-timeout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10000&lt;/span&gt;
      &lt;span class="na"&gt;maximum-pool-size&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt;
      &lt;span class="na"&gt;minimum-idle&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
      &lt;span class="na"&gt;pool-name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;WriteHikariPool&lt;/span&gt;

  &lt;span class="na"&gt;datasource-read&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;driver-class-name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org.postgresql.Driver&lt;/span&gt;
    &lt;span class="na"&gt;jdbc-url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;jdbc:postgresql://localhost:5433/demo&lt;/span&gt;
    &lt;span class="na"&gt;username&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;postgres_user_for_db_read'&lt;/span&gt;
    &lt;span class="na"&gt;password&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;you_password'&lt;/span&gt;
    &lt;span class="na"&gt;platform&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;postgresql&lt;/span&gt;
    &lt;span class="na"&gt;hikari&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;idle-timeout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10000&lt;/span&gt;
      &lt;span class="na"&gt;maximum-pool-size&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt;
      &lt;span class="na"&gt;minimum-idle&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
      &lt;span class="na"&gt;pool-name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ReadHikariPool&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;as you see I have 2 data-source as: datasource-write and datasource-read with their own credentials.&lt;/p&gt;

&lt;p&gt;DataSource Configurations for WriteDB:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Configuration&lt;/span&gt;
&lt;span class="nd"&gt;@ConfigurationProperties&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"spring.datasource-write"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nd"&gt;@EnableTransactionManagement&lt;/span&gt;
&lt;span class="nd"&gt;@EnableJpaRepositories&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;entityManagerFactoryRef&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"entityManagerFactoryWrite"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;transactionManagerRef&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"transactionManagerWrite"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;basePackages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"com.ehsaniara.multidatasource.repository.writeRepository"&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;DataSourceConfigWrite&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="nc"&gt;HikariConfig&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="no"&gt;PERSISTENCE_UNIT_NAME&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"write"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="nd"&gt;@Bean&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;HikariDataSource&lt;/span&gt; &lt;span class="nf"&gt;dataSourceWrite&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;HikariDataSource&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="nd"&gt;@Bean&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;LocalContainerEntityManagerFactoryBean&lt;/span&gt; &lt;span class="nf"&gt;entityManagerFactoryWrite&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;
            &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="nc"&gt;HikariDataSource&lt;/span&gt; &lt;span class="n"&gt;dataSourceWrite&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;LocalContainerEntityManagerFactoryBean&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{{&lt;/span&gt;
            &lt;span class="n"&gt;setDataSource&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataSourceWrite&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setPersistenceProviderClass&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;HibernatePersistenceProvider&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setPersistenceUnitName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="no"&gt;PERSISTENCE_UNIT_NAME&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setPackagesToScan&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="no"&gt;MODEL_PACKAGE&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setJpaProperties&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="no"&gt;JPA_PROPERTIES&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}};&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="nd"&gt;@Bean&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;PlatformTransactionManager&lt;/span&gt; &lt;span class="nf"&gt;transactionManagerWrite&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;EntityManagerFactory&lt;/span&gt; &lt;span class="n"&gt;entityManagerFactoryWrite&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;JpaTransactionManager&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;entityManagerFactoryWrite&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;DataSource Configurations for ReadDB:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Configuration&lt;/span&gt;
&lt;span class="nd"&gt;@ConfigurationProperties&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"spring.datasource-read"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nd"&gt;@EnableTransactionManagement&lt;/span&gt;
&lt;span class="nd"&gt;@EnableJpaRepositories&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;entityManagerFactoryRef&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"entityManagerFactoryRead"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;transactionManagerRef&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"transactionManagerRead"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;basePackages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"com.ehsaniara.multidatasource.repository.readRepository"&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;DataSourceConfigRead&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="nc"&gt;HikariConfig&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="no"&gt;PERSISTENCE_UNIT_NAME&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"read"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;


    &lt;span class="nd"&gt;@Bean&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;HikariDataSource&lt;/span&gt; &lt;span class="nf"&gt;dataSourceRead&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;HikariDataSource&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="nd"&gt;@Bean&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;LocalContainerEntityManagerFactoryBean&lt;/span&gt; &lt;span class="nf"&gt;entityManagerFactoryRead&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;
            &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="nc"&gt;HikariDataSource&lt;/span&gt; &lt;span class="n"&gt;dataSourceRead&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;LocalContainerEntityManagerFactoryBean&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{{&lt;/span&gt;
            &lt;span class="n"&gt;setDataSource&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataSourceRead&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setPersistenceProviderClass&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;HibernatePersistenceProvider&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setPersistenceUnitName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="no"&gt;PERSISTENCE_UNIT_NAME&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setPackagesToScan&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="no"&gt;MODEL_PACKAGE&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;setJpaProperties&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="no"&gt;JPA_PROPERTIES&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}};&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="nd"&gt;@Bean&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;PlatformTransactionManager&lt;/span&gt; &lt;span class="nf"&gt;transactionManagerRead&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;EntityManagerFactory&lt;/span&gt; &lt;span class="n"&gt;entityManagerFactoryRead&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;JpaTransactionManager&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;entityManagerFactoryRead&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read and Write repositories should be in a separated packages :&lt;/p&gt;

&lt;p&gt;Write: &lt;strong&gt;com.ehsaniara.multidatasource.repository.writeRepository&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Read: &lt;strong&gt;com.ehsaniara.multidatasource.repository.readRepository&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;you also need to set:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="no"&gt;MODEL_PACKAGE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"com.ehsaniara.multidatasource.model"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="nc"&gt;Properties&lt;/span&gt; &lt;span class="no"&gt;JPA_PROPERTIES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Properties&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{{&lt;/span&gt;
    &lt;span class="n"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"hibernate.dialect"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"org.hibernate.dialect.PostgreSQL10Dialect"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"hibernate.hbm2ddl.auto"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"update"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"hibernate.ddl-auto"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"update"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"show-sql"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"true"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the actual logic are in the service layer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Service&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;CustomerServiceImpl&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="nc"&gt;CustomerService&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="nc"&gt;CustomerReadRepository&lt;/span&gt; &lt;span class="n"&gt;customerReadRepository&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="nc"&gt;CustomerWriteRepository&lt;/span&gt; &lt;span class="n"&gt;customerWriteRepository&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;CustomerServiceImpl&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;CustomerReadRepository&lt;/span&gt; &lt;span class="n"&gt;customerReadRepository&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;CustomerWriteRepository&lt;/span&gt; &lt;span class="n"&gt;customerWriteRepository&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;customerReadRepository&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;customerReadRepository&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;customerWriteRepository&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;customerWriteRepository&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;Optional&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Customer&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;getCustomer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Long&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;customerReadRepository&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;findById&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;Customer&lt;/span&gt; &lt;span class="nf"&gt;createCustomer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Customer&lt;/span&gt; &lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

        &lt;span class="nc"&gt;Assert&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;notNull&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"Invalid customer"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="nc"&gt;Assert&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;isNull&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getId&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="s"&gt;"customer id should be null"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="nc"&gt;Assert&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;notNull&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getName&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="s"&gt;"Invalid customer name"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;customerWriteRepository&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;save&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;Customer&lt;/span&gt; &lt;span class="nf"&gt;updateCustomer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Customer&lt;/span&gt; &lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

        &lt;span class="nc"&gt;Assert&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;notNull&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"Invalid customer"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="nc"&gt;Assert&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;notNull&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getId&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="s"&gt;"Invalid customer id"&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;customerWriteRepository&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;save&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now if run this line you create customer in DB1:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl -H "Content-Type: application/json" --request POST --data '{"name":"Jay"}'   http://localhost:8080/customer
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;OR&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl -H "Content-Type: application/json" --request PUT --data '{"id":1 , "name":"Jay ehsaniara"}'   http://localhost:8080/customer
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But if you run this line you getting data from DB2:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; curl --request GET  http://localhost:8080/customer/1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note: you need to insert customer manually in DB2 since it has no pre customer. and we haven't setup Postgres Replication yet&lt;/p&gt;

</description>
      <category>java</category>
      <category>springboot</category>
      <category>datasources</category>
      <category>jpa</category>
    </item>
  </channel>
</rss>
