DEV Community: Marley Spoon

The Four Principles of Design Thinking in Software Architecture

João Hélio — Wed, 12 Feb 2025 09:20:42 +0000

Design Thinking is less a process and more a way to think about problems and solutions from the people's perspective and how the architecture affects the stakeholders.

In this post, I’m going to highlight the four principles of Design Thinking and how they can be applied to software development to guide our design activities.

🔹 Design for Humans
We design software for people and with people, in the end, software development is an intensely social and collaborative activity. The idea to create a software architecture isolated from the team is a myth.

As we design a system, we work closely with our team, fostering mutual respect by actively listening, assuming positive intent, and embracing human-centered design principles.

Empathizing with the humans who directly and indirectly interact with the architecture makes us a better designer, communicators, and leaders.

🔹 Preserve Ambiguity
Ambiguity in engineering is dangerous. Once we've made a design decision, we must share it with precision and clarity. However, design decisions that do not directly influence a quality attribute or reduce risk threatening our ability to deliver software are more about detailed design than architecture.

Such decisions can safely be left open for downstream designers to settle outside the architecture.

🔹 Design is Redesign
The redesign rule encourages us to think about what we already know by exploring patterns and past designs. As time goes on and as we build more software, our institutional knowledge about how to design great software improves. Other teams have probably seen a problem similar to the one you are facing currently. Hopefully, someone documented a pattern you can use as a starting point for your architecture.

When designing software architectures, we'll spend more time refining existing designs than creating new ones. One of the least effective ways to design an architecture is to ignore the software system that came before us.

🔹 Make the Architecture Tangible
While the structures in the architecture can exist in code, this does not make the architecture any more tangible.

Code is difficult to read and does not make discussions about quality attributes, coarse-grained components, design rationale, or the consequences of our decisions any easier.

If we want to share an architecture with others, then we need to make it real in a way that code by itself will not allow.

The tangible rule is closely related to the human rule. Humans must be able to relate to ideas to internalize them.

Reference
Design It: From Programmer to Software Architect by Michael Keeling

What are your thoughts? Let’s discuss it! 👇

Thinking in Abstractions

João Hélio — Wed, 09 Oct 2024 09:57:49 +0000

A quick fix in the code can solve many issues, but it may create complexity that becomes difficult to maintain over time. This happens as more and more rules are added to address specific problems within a general context. Let's look at a use case and explore some potential solutions.

Use Case

Imagine you have a general Order class, but now the business requires handling two specific cases. The Halloween Order must contain at least ten line items, while the Christmas Order can be empty but must include at least one add-on.

First Try: `IF-ELSE` Statement

We can attempt to handle this with conditionals like the following:

class Order
  attr_accessor :line_items, :plan, :type, :add_ons

  def initialize(line_items, plan = 'Basic', type = 'Standard', add_ons = [])
    @line_items = line_items
    @plan = plan
    @type = type
    @add_ons = add_ons
  end

  def validate
    if @type == 'Standard'
      raise 'Total amount must be greater least one line item' if @line_items.empty?
    end

    if @type == 'Halloween'
      raise 'Halloween order must have at least ten line items' if @line_items.size < 10
    end

    if @type == 'Christmas'
      raise 'Christmas order must have at least one add-on' if @add_ons.empty?
    end
  end
end

# Example usage
halloween_order = Order.new([], 'Premium', 'Halloween')
halloween_order.validate
=> `validate': Halloween order must have at least ten line items (RuntimeError)

christmas_order = Order.new([], 'Enterprise', 'Christmas')
christmas_order.validate
=> `validate': Total amount must be greater least one line item (RuntimeError)

Pros

Simplicity in Small-Scale Scenarios: using if-else statements can be the quickest way to implement logic.
Fewer Lines of Code: For simple conditional logic may result in fewer lines of code at first.
No Need for Overhead in Basic Use Cases: In situations where we don’t expect many changes or extensions to the logic, if-else can avoid unnecessary overhead.

Cons

Mixed Responsibilities: The Order class is now responsible for both general order logic and the specific rules for Halloween and Christmas orders.
Difficult to Extend: Adding a new order type means modifying the core Order class, which introduces risk and reduces flexibility.
Harder to Test: We can end up having a test that is hard to understand when specific rules are incorporated.

Second try: Abstraction Through Inheritance

We can create specialized classes that capture the architectural intent. By abstracting common behaviour into a base class and allowing each type to manage its own rules.

class Order
  attr_accessor :line_items, :plan, :add_ons

  def initialize(line_items, plan = 'Basic', add_ons = [])
    @line_items = line_items
    @plan = plan
    @add_ons = add_ons
  end

  def validate
    raise 'Total amount must be greater least one line item' if @line_items.empty?
  end
end

class HalloweenOrder < Order
  def initialize(line_items, plan = 'Basic', add_ons = [])
    super(line_items, plan, add_ons)
  end

  def validate
    raise 'Halloween order must have at least ten line items' if @line_items.size < 10
  end
end

class ChristmasOrder < Order
  def initialize(line_items, plan = 'Basic', add_ons)
    super(line_items, plan, add_ons)
  end

  def validate
    raise 'Christmas order must have at least one add-on' if @add_ons.empty?
  end
end

# Example usage
halloween_order = HalloweenOrder.new([], 'Premium')
halloween_order.validate                                 
=> `validate': Halloween order must have at least ten line items (RuntimeError)

christmas_order = ChristmasOrder.new([], 'Enterprise')
christmas_order.validate
=> `validate': Christmas order must have at least one add-on (RuntimeError)

Pros

Separation of Concerns: The base Order class only handles shared behaviour. The specific rules for HalloweenOrder and ChristmasOrder are handled in their respective classes.
Scalability: Adding a new order type (e.g., EasterOrder, BlackFridayOrder) is easy.
Easier to Test: Testing new use cases doesn't require a shared setup.

Cons

Overhead for Simple Use Cases: When the use case is simple, setting up multiple classes and abstractions can introduce unnecessary complexity.
Over-Engineering: YAGNI ("You Aren’t Gonna Need It") we might introduce abstractions for future possibilities that never materialize.
More Classes to Maintain: Abstracting logic into specialized classes can increase the number of files and classes, which means more "moving parts".

Conclusion

The patch and the module abstraction methods have their place in software design. The key is to assess the complexity and evolution of the system.

Start simple and refactor when complexity arises. This balance ensures the architecture remains maintainable and adaptable without becoming overly complicated.

Retro Guide

João Hélio — Mon, 20 Nov 2023 09:20:37 +0000

Introduction

The sprint retrospective is a moment for our team to reflect on the past sprint, celebrate successes, and identify areas for improvement. By fostering a culture of continuous learning, we aim to enhance our work process and achieve even greater success in the future. To achieve this, strategic planning is essential, ensuring that the retrospective becomes a well-orchestrated effort toward sustained improvement and growth.

Directives

At the heart of our retrospective is the understanding that everyone did their best given the circumstances. We value cooperation, recognizing that our unique strengths are amplified when we work together. Hope and trust emerge through engagement, and we seize this opportunity to unite around a shared vision for the future.

Context

Gaining insight into the project context provides a clear understanding of our position in the project, making it particularly valuable for the sprint retrospective. If time constraints prevent context discussion during other events like sprint review or planning, consider incorporating it as an introduction to the retrospective.

Here are some aspects of the context that might be relevant to a retrospective:

Start or End of a Project

Whether the retrospective marks the beginning or end of a project, the context would include specific project goals, team changes, initial expectations, and achievements.

Team Changes

If there have been significant changes to the team composition (members joining or leaving), the context may include how these changes impacted team dynamics and performance.

Implementation of New Processes

If the team has recently adopted new processes or tools, context can address the effectiveness of these changes and how they have influenced workflow.

Development Cycle

Timing in the development cycle iteration (such as a specific sprint) is essential. Discussions can focus on goals met, user stories delivered, and obstacles faced during this sprint.

External Events

External events, such as changes in customer demands, industry updates, or unexpected challenges, provide crucial context for understanding how the team responded to these external factors.

Performance Metrics

Quantitative metrics such as team velocity, user story completion rate, or code quality can be part of the context, providing an objective basis for analysis.

Customer Feedback

If there was customer feedback during or after the sprint, that feedback becomes part of the context, helping the team understand how their deliverables are perceived.

Organizational Objectives

Contextualizing the retrospective about the organization's broader goals helps align the team's priorities with the company's overall goals.

Evaluation of Training or Culture Changes

If the team participated in training or if the organization is going through cultural changes, this context is crucial to evaluate how these initiatives are being absorbed and applied.

Focus on the Current Sprint

Direct the discussion to the most recent sprint, ensuring that observations and learnings are specific to the team's current situation.

Specific Goal Orientation

This allows a team to focus on specific areas that need attention and improvement.

Connection to Long-Term Goals

The retrospective should be seen as part of a continuous improvement process. Timing/context helps link the discussion to the long-term goals of the team and organization.

Identification of Trends Over Time

You can identify trends over multiple sprints, which can lead to relevant insights into performance patterns and consistent areas for improvement.

Participant Preparation

For our retrospective to be effective, the team members are encouraged to reflect on specific questions before the meeting to facilitate a more insightful discussion:

What worked well:

Standout moments in the sprint.

Example: "The continued refinement of the Rewards topic was a highlight, allowing us to define the goals of the domain better."

Effective practices or processes.

Example: "Adopting the clean architecture concepts has helped us to create a more scalable system."

Exceptional collaboration between team members.

Example: "Close collaboration between developers and PMs during the Q&A phase facilitated rapid identification and resolution of bugs."

Individual skills that contributed to success.

Example: "John's problem-solving skills were instrumental in overcoming unexpected challenges during development."

Aspects of the work that led to overall success.

Example: "Clarity in user stories helped avoid misunderstandings and allowed smoother implementation."

Positive changes compared to the previous sprint.

Example: "Implementing excellent_migrations tool reduced the bugs to zero when creating a new migration."

Each team member's main contributions.

Example: "Maria effectively led the resolution of a critical problem, demonstrating her ability to deal with challenging situations."

What can be improved:

Main obstacles or challenges faced.

Example: "Integration with the legacy API system was more complicated than expected, causing significant delays."

Communication difficulties negatively impact the team.

Example: "The lack of regular updates on task progress has confused me about priorities. When we say 'I'm working on it and don't have any blockers' on Dailies meeting, we lose the opportunity to receive feedback and share knowledge."

Aspects of the work process that can be optimized.

Example: "Reviewing milestones and deadlines regularly can give us evidence to adapt to the current tasks."

Recurring issues that need addressing.

Example: "Dependence on a single specialist for certain critical tasks has caused bottlenecks. We must diversify knowledge within the team."

Gaps in the team’s skills or knowledge.

Example: "The lack of expertise in a new technology impacted the efficient implementation of a feature. We must seek training."

Handling external pressures or tight deadlines.

Example: "Deadline pressure has affected code quality. We should explore strategies to maintain an appropriate balance between speed and quality."

Tasks or activities that did not contribute significantly.

Example: "Allocating extensive resources to infrequently requested functionality may have been unnecessary. We must evaluate the value before initiating such tasks."

Checkin

Collect the participants' sentiments. This sets the tone for a focused and constructive discussion.

Continuous Improvement

Engage in data collection, emotional expression, and acknowledgment of positive aspects. Explore opportunities for enhancement to commit to continuous improvement.

Filtering

Prioritize discussion topics to generate insights within the available time efficiently. Streamline the conversation to focus on the most crucial areas.

Checkout

As the meeting concludes, assign responsibilities for action items identified during the retrospective. Clearly define who will be responsible for each task.

Conclusion

In summary, the retrospective process is designed to promote a culture of continuous improvement within our team. By encouraging thoughtful participant preparation, considering relevant context, and fostering open communication, we aim to glean valuable insights for enhancing our work processes. Through these efforts, we aspire to create a collaborative environment that consistently strives for excellence.

References

Fun Retrospectives: A diverse collection of activities and techniques for engaging retrospectives.
The Retrospective Handbook - A guide for agile teams by Patrick Kua: A practical guide for agile teams running effective retrospectives.
Agile Retrospectives - Making Good Teams Great by Esther Derby, Diana Larsen, Ken Schwaber: A comprehensive overview of conducting effective agile retrospectives.

The Request and Response Model in the Clean Architecture

João Hélio — Thu, 31 Aug 2023 10:00:27 +0000

Introduction

One of the key concepts within the Clean Architecture is the Request and Response model, which helps to separate the core application logic from external concerns such as user interfaces and frameworks. In this article, we'll explore the Request and Response model and demonstrate its implementation using Ruby.

The Request and Response Model

The Request and Response model is a pattern that promotes a clear separation between the inputs and outputs of the application. By doing so, the architecture becomes more adaptable to changes and easier to test, as the core logic is not tightly coupled to any specific framework or external component.

Request

A request in the Clean Architecture represents the input to a use case or an operation within the application. It encapsulates the necessary data required for the operation to be executed. Requests are usually simple data structures and do not contain any business logic. They act as a boundary between the external world (such as UI) and the application's core.

Response

On the other hand, a response represents the output of a use case or operation. It contains the result of the operation, including any data that needs to be presented to the user or propagated to other parts of the application. Similar to requests, responses are also typically simple data structures.

Implementing

Let's consider a simple example of a blog application. We'll implement the Request and Response model for a use case where a user wants to create a new blog post.

In the following example, we define PostRequest as a Dry::Validation::Contract and PostResponse using Dry::Struct. The CreatePost use case takes a request object, processes the business logic, and returns a response object.

require 'dry/struct'
require 'dry/validation'

module Blog
  module UseCases
    module Posts
      class PostRequest < Dry::Validation::Contract
        params do
          required(:title).filled(:string)
          required(:content).filled(:string)
          required(:author_id).filled(:integer)
        end
      end

      class PostResponse < Dry::Struct
        attribute :success, Types::Bool
        attribute :message, Types::String
        attribute :post_id, Types::Integer.optional
      end

      class CreatePost
        def call(request)
          validation_result = PostRequest.new.call(request)

          if validation_result.success?
            # Business logic to create a new post
            # ...

            # Return a response
            PostResponse.new(success: true, message: 'Post created successfully', post_id: 1)
          else
            # Return a response with validation errors
            PostResponse.new(success: false, message: 'Validation errors', post_id: nil)
          end
        end
      end
    end
  end
end

# Example usage
request = Blog::UseCases::Posts::PostRequest.new(
  title: 'Sample Post',
  content: 'This is the content of the post.',
  author_id: 123
)

use_case = Blog::UseCases::Posts::CreatePost.new
response = use_case.call(request)

puts response.message

Let's see how we can write a test for the CreatePost use case:

RSpec.describe Blog::UseCases::Posts::CreatePost do
  let(:use_case) { described_class.new }

  context 'when the request is valid' do
    let(:valid_request) do
      Blog::UseCases::Posts::CreatePostRequest.new(
        title: 'Sample Post',
        content: 'This is the content of the post.',
        author_id: 123
      )
    end

    it 'creates a new post' do
      response = use_case.call(valid_request)
      expect(response.success).to eq(true)
      expect(response.message).to eq('Post created successfully')
      expect(response.post_id).to eq(1) # Replace with your expected post ID
    end
  end

  context 'when the request is invalid' do
    let(:invalid_request) do
      Blog::UseCases::Posts::CreatePostRequest.new(
        title: '',
        content: 'This is the content of the post.',
        author_id: nil
      )
    end

    it 'returns validation errors' do
      response = use_case.call(invalid_request)
      expect(response.success).to eq(false)
      expect(response.message).to eq('Validation errors')
      expect(response.post_id).to be_nil
    end
  end
end

To explore the full capabilities of the dry-rb ecosystem check the website.

Benefits of the Request and Response Model

By separating requests and responses, we achieve a clear boundary between external concerns and core business logic and since requests and responses are simple data structures, testing becomes straightforward, and we can easily write unit tests for each use case.

The Clean Architecture, coupled with the Request and Response model, allows us to change external components or frameworks without affecting the core logic.

In conclusion, the Request and Response model is a powerful concept within the Clean Architecture that promotes separation of concerns and maintainability.

References

"Clean Architecture: A Craftsman's Guide to Software Structure and Design" by Robert C. Martin

Understanding Bounded Contexts in Ruby

João Hélio — Thu, 17 Aug 2023 09:33:28 +0000

Introduction

Bounded contexts have a crucial role in managing complexity, maintaining a clear separation of concerns, and ensuring that different parts of a system can communicate effectively without causing conflicts or confusion. Bounded contexts are a fundamental concept in Domain-Driven Design (DDD) and are particularly relevant when building complex systems.

In this article, the bounded contexts will be explored using a practical example in Ruby, where we'll demonstrate how to consume and wrap a Plan object between two different contexts: the Plan Management context and the Sales context.

Understanding Bounded Contexts

A bounded context is a conceptual boundary within which a particular domain model is defined and applicable. Different parts of a software system may have different understandings of specific terms, rules, and behaviors. Bounded contexts help avoid ambiguity and conflicts by isolating other models within their own context, making sure they remain consistent and meaningful within that context.

In our example scenario, imagine a software application for a company that offers subscription plans to its customers. There are two primary bounded contexts: Plan Management and Sales.

Plan Management Context: This context is responsible for creating, updating, and managing subscription plans. It focuses on the administrative aspects of plan creation and modification.
Sales Context: This context is responsible for handling customer interactions, including the sale of subscription plans. It deals with pricing, availability, and customer-specific details.

Consuming and Wrapping Plan Objects

Let's dive into the example scenario to understand how bounded contexts work in Ruby. We'll begin by creating a simplified Plan class that holds basic plan information. Then, we'll demonstrate how to consume and wrap a Plan object between the two contexts.

# Plan class in the Plan Management context
module PlanManagement
  class Plan
    attr_reader :id, :name, :price

    def initialize(id, name, price)
      @id = id
      @name = name
      @price = price
    end

    def update_price(new_price)
      @price = new_price
    end
  end
end

# Plan class in the Sales context
module Sales
  class Plan
    attr_reader :customer, :plan_dto

    def initialize(customer, plan_dto)
      @customer = customer
      @plan_dto = plan_dto
    end

    def calculate_discount
      (customer.credit / plan_dto.price) * 100
    end

    def name
      plan_dto.name.upcase
    end

    def price
      plan_dto.price - customer.credit
    end
  end
end

In this example, we have two classes defined in different contexts, the Plan class in the Plan Management context and the Plan in the Sales context. Each class is tailored to its respective context's needs.

To maintain the separation between the contexts and ensure smooth communication, we need to wrap Plan objects when transitioning between contexts. Let's illustrate how this can be done:


# Plan Management context
plan = PlanManagement::Plan.new(1, "Basic Plan", 10.0)

# Sales context
customer = Customer.new("John Doe")
sales_plan = Sales::Plan.new(customer, plan)

# Example usage in the Sales context
discount = sales_plan.calculate_discount
puts "Customer #{customer.name} gets a discount of #{discount}% on the #{sales_plan.name} plan."

puts "Plan details:"
puts "Name: #{sales_plan.name}"
puts "Price: $#{sales_plan.price}"

In this example, the Sales::Plan class provides a bridge between the Sales context and the Plan Management context. It wraps the Plan object from the Plan Management context and exposes only the methods relevant to the Sales context, such as calculating discounts based on a customer's profile.

Conclusion

By defining boundaries around different domain models and using wrapper classes when transitioning between contexts, we can maintain consistency, reduce conflicts, and promote a better understanding of how other parts of the system interact. In our Ruby example, we've demonstrated how bounded contexts can be applied to a Plan object, showcasing the practical benefits of this concept in real-world software development.

References

Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans This book is often considered the authoritative source on Domain-Driven Design (DDD) and introduces the concept of bounded contexts along with various other DDD principles.
Bounded Context - Martin Fowler's blog is a valuable resource for software design patterns and concepts. His article on bounded contexts provides a comprehensive explanation with examples.
Link: https://martinfowler.com/bliki/BoundedContext.html

Taming Time: how to run XTDB in production

Alex Kuznetsov — Wed, 19 Jul 2023 11:13:57 +0000

In the previous articles, we explored the concept of bitemporality and discussed how to get started with XTDB, a bitemporal immutable database. Now, let’s dive into the technical details of deploying XTDB and running it in production. This blog post aims to provide valuable insights and considerations to keep in mind during this process.

XTDB 1

Before we proceed, it’s important to note that the experiences shared in this article are based on working with XTDB version 1.21.
It’s worth mentioning that your experience may vary, especially with the introduction of XTDB 2.0 and subsequent versions.

Deploying XTDB

Deploying XTDB in a production environment offers several options, each with its advantages and considerations:

run as a part of your JVM application
run separately, but on the same server together with your application
run on a standalone node or in a separate container
run a cluster of XTDB nodes

To achieve a resilient and scalable setup, I recommend running a cluster of XTDB nodes, with each node deployed as a separate Docker container managed by Kubernetes. This allows easy orchestration, automatic scaling, and simplified management of the XTDB cluster.

Containerised XTDB

In this section, we will explore how to build and run a Docker container with a custom configured XTDB application (Clojure project) in a Kubernetes cluster. However, in case you don’t need to have a custom build and can simply use a standalone Docker image, you can skip right to the "Authentication" section.

Uberjar

If we want to run the XTDB in Docker the most fitting way to run it in a container would be to compile our preconfigured XT application into a single “uberjar” file.

This can be done by employing Uberdeps in our Clojure project:

    ; deps.edn
    :aliases
     {:uberdeps {:replace-deps {uberdeps/uberdeps {:mvn/version "1.1.0"}}
                 :replace-paths []
                 :main-opts ["-m" "uberdeps.uberjar"]}}

Once you have it installed, you can compile your project into a single Uberjar file:

clj -M:uberdeps

Dockerfile

We want our XTDB Docker image to be as lightweight as possible, so the best approach would be to have a multi-stage build image that builds and runs the Uberjar on the selected JVM image:

    # Build clojure uberjar
    FROM clojure:openjdk-17-tools-deps-alpine AS BUILD

    WORKDIR /xtdb
    COPY . /xtdb
    RUN apk add --no-cache libstdc++

    RUN clojure -M:uberdeps

    # Copy and run uberjar
    FROM openjdk:17-alpine3.14
    WORKDIR /usr/local/lib/xtdb

    RUN apk --no-cache add bash libstdc++

    COPY --from=BUILD /xtdb/resources /usr/local/lib/xtdb/resources
    COPY --from=BUILD /xtdb/target .

    ENV MALLOC_ARENA_MAX=2
    CMD ["java", "-cp", "xtdb.jar", "clojure.main", "-m", "xtdb.core"]

    EXPOSE 3000

Graceful shutdown

Another requirement for running XT in Kubernetes is to gracefully shut down, e.g., on killing containers or Kubernetes deployment restart. To ensure that, we have to change our xtdb.clj file by adding a SIGTERM signal handler:

    ;; Stop the system on SIGTERM
    (with-handler :term
      (log/info "Caught SIGTERM, quitting")
      (.close @xt-node)
      (log/info "All components shut down")
      (System/exit 0))

Authentication

Since we run our XTDB separately from our application containers, we might want to ensure that the requests from services to the database are properly authenticated.

In order to ensure that we need to change the way we start XTDB by providing JWKS (JSON Web Token key set) as an environment variable to the XTDB node:

    ; core.clj
    (def config
      {:xtdb.http-server/server {:port 3000
                                 :jwks (System/getenv "XTDB_JWKS")} ...

The next step is to send the compatible JWT token from your application requests:

    {"Authorization", "Bearer #{your_jwt}"}

Kubernetes

As we’ve decided to go with the running XT nodes as Docker containers in a Kubernetes cluster, we need to prepare Kubernetes manifests for that purpose.

The simplest way to achieve that is to create a Kubernetes stateful set of multiple XT containers:

    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: xtdb
      labels:
        app.kubernetes.io/name: xtdb
    spec:
      serviceName: xtdb-headless
      replicas: 3
      selector:
        matchLabels:
          app.kubernetes.io/name: xtdb
      template:
        metadata:
          labels:
          app.kubernetes.io/name: xtdb
        spec:
          terminationGracePeriodSeconds: 30
          containers:
            - name: xtdb
              image: $REGISTRY/xtdb:1.21.0
              imagePullPolicy: Always
              ports:
                - containerPort: 3000
              livenessProbe:
                tcpSocket:
                  port: 3000
                periodSeconds: 10
                initialDelaySeconds: 120
                timeoutSeconds: 15
              readinessProbe:
                exec:
                  command:
                    - bash
                    # custom readiness check script
                    - scripts/readiness.sh
                initialDelaySeconds: 30
                periodSeconds: 15
              envFrom:
                - secretRef:
                    name: xtdb-secrets

As for the configuration, you can create a configmap with environment variables required for configuring XTDB and a secrets resource for providing secrets to your XT nodes.

Running XT in production

XTDB is an unbundled database which means that it has a lot of components that can be swapped or changed — and it might work with different technologies and other databases.

In general, it consists of 3 parts:

Transaction log
Document store
Index store

We had an experience using PostgreSQL and JDBC adapter for our XTDB setup, as well as experimenting with Kafka for transaction log — and using RocksDB as an index storage.

However, there are many more other ways and modules to setup the database — you can find them in the documentation.

JDBC

The transaction log and document store are considered to be golden stores in XTDB — which means that they should be reliably persisted, unlike the index storage that can be rebuilt from scratch on node restart.

XT supports JDBC (Java Database Connectivity) which allows us to connect to various SQL databases like PostgreSQL, MySQL, SQLite, and others.
In our example, we were using PostgreSQL + JDBC for both transaction log and document store.

To use PostgreSQL and JDBC together, you have to provide these modules in your deps.edn first:

:deps {org.postgresql/postgresql {:mvn/version "42.2.18"}
        com.xtdb/xtdb-jdbc {:mvn/version "1.21.0"}
       ...

If you want to use environment variables for connecting to PostgreSQL from the XTDB deployment you can also pass them in core.clj file:

    (def db-spec
      {:host (System/getenv "POSTGRES_HOST")
       :port (get (System/getenv) "POSTGRES_PORT" "5432")
       :dbname (System/getenv "POSTGRES_DB")
       :user (System/getenv "POSTGRES_USER")
       :password (System/getenv "POSTGRES_PASSWORD")})

    (def config
      {:xtdb.jdbc/connection-pool {:dialect {:xtdb/module 'xtdb.jdbc.psql/->dialect}
                                   :pool-opts {:maximumPoolSize 10}
                                   :db-spec db-spec}
       :xtdb/tx-log {:xtdb/module 'xtdb.jdbc/->tx-log
                     :connection-pool :xtdb.jdbc/connection-pool}
       :xtdb/document-store {:xtdb/module 'xtdb.jdbc/->document-store
                             :connection-pool :xtdb.jdbc/connection-pool}})

That way we can also re-use the same connection pool for both transaction and document store.

Kafka

However, the recommended option (and also most often used in production) is to leverage Kafka for the transaction store.

During node restarts (e.g. on new deployments) XTDB has to rebuild the transaction log from zero or the latest saved checkpoint, which means reading the whole transaction table in PostgreSQL — and in our experience, this process was slower than expected.

Kafka seems to be a better fit for the very purpose of the transaction log because:

it’s basically a log of events
we only need to use just one partition and one topic
the transactions can be consumed very quickly

Thus, the most optimal and performant setup for us looked like:

using Kafka as a transaction log
using JDBC and relational database as a document store
using RocksDB as an index store

That way, we can ensure that the transactions can be created quickly, the log can be re-consumed fast, and the document storage is performant enough and resilient.

Checkpoints

If we want to rebuild the query indices (e.g. on node restart), XT might need to replay the transaction log — which sometimes might be not so fast especially if you have a long history of changes.

As we run our XTDB in a cluster, it’s vital that the readiness time — when the XT instance is ready to serve DB requests — is as low as possible.
Fortunately, XTDB has a solution for that problem called checkpoints.

Right now, there are three ways to persist the local query indices state:

local files (using Java’s NIO file system)
AWS S3
GPC’s cloud storage

AWS S3

In our case, we’ve decided to go with the AWS setup in order to have a centralised and already configured storage for the checkpoints.
However, it also requires some additional dependencies to be installed:

    ; deps.edn
     :deps {com.xtdb/xtdb-s3 {:mvn/version "1.21.0"}
            software.amazon.awssdk/aws-core {:mvn/version "2.10.91"}

Additional setup in the node configuration is also required.

As we’ve experienced some requests to S3 taking a long time, we’ve also decided to build a custom AWS S3 HTTP client:

    (defn- make-s3-client
      "Increases timeouts for AWS S3 HTTP calls"
      []
      (let [timeout (Duration/ofSeconds 30)
            http-client-builder (->
                                 (NettyNioAsyncHttpClient/builder)
                                 (.connectionAcquisitionTimeout timeout)
                                 (.connectionTimeout timeout))]
        (-> (S3AsyncClient/builder)
            (.httpClientBuilder http-client-builder)
            .build)))

    (def checkpoint-name (get (System/getenv) "CHECKPOINT_NAME" ""))

    (def checkpoint-config
      ; Checkpoints are not enabled on a local machine where we don't have the env
      (if (str/blank? checkpoint-name)
        {}
        {:xtdb/module 'xtdb.checkpoint/->checkpointer
         :store
         {:xtdb/module 'xtdb.s3.checkpoint/->cp-store
          :bucket checkpoint-name
          :configurator (fn [_] (reify S3Configurator (makeClient [_this] (make-s3-client))))}
         :Keep-dir-on-close? false
         :approx-frequency (Duration/ofHours 2)}))

Once configured, XT will persist the current index state to S3 every 2 hours. One might want to adjust S3’s bucket policy so it archives or removes the obsolete checkpoints files.

Caveats

Memory consumption

JVM-based applications tend to consume quite a significant amount of memory budget — in our case, running XT with allowed 4GB of memory wasn’t always enough so we’ve decided to increase the memory limits in Kubernetes up to 8 gigabytes.

Another consideration we’ve observed is that the vertical scaling works better for an XTDB cluster — unlike the horizontal scaling, where we need to wait until the new node restores from the checkpoints or processes the transactions log.

RocksDB tuning

RocksDB is being used by XT as an index store, and as the result, it might consume quite a significant amount of resources.
In order to avoid possible issues with the memory budgeting, it’s recommended to set RocksDB block cache to 1/3 of available memory which can be done in XTDB configuration.

Readiness probes

Depending on your technology stack that you use for XTDB deployment, consuming the transaction log even with the checkpoints feature enabled can take some time — and even though the starting node is able to handle REST API requests, they won’t be processed until the node finishes the consumption.

To avoid that, you might need to check the difference between the last submitted and last completed transactions, e.g. from a bash script:

    #!/bin/bash
    # scripts/readiness.sh: A script that compares the latest submitted and indexed transactions

    set -e
    THRESHOLD=1000

    # Assumes that you have jq and curl installed
    submitted_tx=`curl http://localhost:3000/_xtdb/latest-submitted-tx -H "Accept: application/json" -f | jq .txId`
    completed_tx=`curl http://localhost:3000/_xtdb/latest-completed-tx -H "Accept: application/json" -f | jq .txId`
    diff=$[submitted_tx - completed_tx]

    if ((diff > THRESHOLD)); then
        echo "Node is not ready"
        exit 1
    else
        echo "Node is ready"
    fi

Load balancing and XT cluster

The index storage is not shared between XTDB nodes — so every node might have a slightly different data representation. To ensure integrity, we might need to use await-tx or sync functions whenever we submit a transaction.

However, when we use REST API in a distributed cluster of nodes, it might be that the load balancer that stands in front of the nodes distributes requests to the database randomly — and when we submit a transaction to one node, we can end up reading data from another, which might have not processed that transaction yet.

If we want to prevent such situation, we might need to implement sticky sessions or use HTTP2 connections between your applications and database nodes.

Conclusion

XTDB embraces the bitemporality concept and provides powerful capabilities of handling your data in an immutable way.

However, this also implies that during your journey with XT, you might face some technical challenges caused by its unbundled database concept — and resolve them by reasoning about the selected components, technology stack, and implications.

The new milestone in XTDB’s development — XTDB 2.0 — looks very promising for us as it has a more flexible and scalable architecture as well as the first-class SQL support — and can be used by PostgreSQL clients.

We look forward to trying out the new version and hope that you’ve enjoyed our series of articles about XT.

Happy hacking, and stay tuned!

Thanks Carsten and Adam for the reviews!

Don't call it `*_id`!

Carsten Zimmermann — Sun, 07 May 2023 10:08:22 +0000

In a distributed system, we often need to store data that we do not own. We might use it as a unique identifier across domains or the system we do own needs to proxy it to yet another service.

In many cases, the origin system exposes it in the same way it is represented internally, for instance: a foreign key name of a relational database:

Fig. 1: local relational table, rev. 1

+----+--------------+----------------------+
| id | name         | backend              |
+----+--------------+----------------------+
| 42 | Marley Spoon | elixir, ruby, python |
+----+--------------+----------------------+

Translated to JSON, we could send it over the wire as follows:

Fig. 2: example JSON payload

{ 
  "id": 42, 
  "name": "Marley Spoon",
  "backend": ["elixir", "ruby", "python"]
}

A service consuming the information may want to add prefixes to scope it for "companies" and store/cache it like so:

Fig. 3: local relational table, rev. 2

+------------+--------------+----------------------+
| company_id | company_name | backend              |
+------------+--------------+----------------------+
| 42         | Marley Spoon | elixir, ruby, python |
+------------+--------------+----------------------+

But here's the problem: as engineers, we look at _id fields and immediately think of it as integers. However, the consuming service has no control over the data it receives and the data type is only assumed.

If you use that distributed ID field as a local foreign key: some external system controls the value and an unforeseen change might break our setup.

Identification

It has proven valuable to us to use the pattern *_identifier to indicate that…

it is some kind of a unique identifier
some other system has control over it

If the *_identifier value is to be stored, it should always be saved as a string type. Almost anything can be coerced into a string, and that way we guarantee that the origin system can choose whatever they want for their unique identifier.

This is particularly true if the origin system decided to move to using UUIDs. A final version of the local relational table above could look like this:

Fig. 4: local relational table, rev. 4

+--------------------------------------+--------------+----
| company_identifier                   | company_name | …
+--------------------------------------+--------------+----
| 328129ae-df4e-4168-94d3-2572b4b343ef | Marley Spoon | …
+--------------------------------------+--------------+----

Payload

If the system exposing the data is controlled by your organisation, we can support this at the source.

It is a common pitfall of API designs, especially RESTful APIs, to expose a resource exactly like you represent it in your database. This makes sense to reduce the cognitive load of the team maintaining the API. However, the data layer will inevitably change, rendering this point moot: the DB representation has to be translated to maintain a stable contract. Why not abstract from the data layer to begin with and name the keys in the payload in a system-agnostic way?

Fig. 5: JSON payload abstracted from persistence

{ 
  "company_identifier": "328129ae-df4e-4168-94d3-2572b4b343ef",
  "company_name": "Marley Spoon",
  "backend": ["elixir", "ruby", "python"]
}

Summary

Use *_identifier instead of *_id fields for externally owned data
Prefer a string type over integer for *_identifier values
Avoid a 1:1-map of your persistence model to your external API

Taming Time: how to install & develop with XTDB

Alex Kuznetsov — Wed, 05 Apr 2023 14:43:22 +0000

In the previous article, we discussed the concept of bitemporality and how it can be used to solve complex architectural problems.

At MarleySpoon, we’ve used XTDB (or ’XT’ for short) for our new order management system (OMS), and discovered a lot of interesting insights about the database itself, the concept of bitemporality, and how developing a project using an immutable, bitemporal database could look like.

In this article, we will be focusing primarily on our development experience (installing, testing with XTDB) from an Elixir application, and we cover more details about deploying, running XT in cluster, and tuning in production in the upcoming article.

What is XTDB

XTDB, or Cross-Time Database, is a distributed and transactional database system designed to handle complex and changing data with ease.
It is based on a bitemporal model, which allows for the tracking of both the valid time and transaction time of data, enabling powerful and flexible querying capabilities.
With XTDB, developers can work with immutable data structures, which simplifies development and improves reliability.
Its graph query language, Datalog, provides a powerful and expressive way to navigate relationships within the data.

As we’ve illustrated before, XT has a lot of benefits:

Bitemporal
Supports retroactive corrections
Document and graph-based
Flexible data schema
Unbundled (can be deployed on top of a lot of other DBs and persistence solutions)
Can be used within JVM or through REST API

As one can see, XT is quite different from most of the widely used SQL and NoSQL databases - and while it provides great benefits for dealing with immutable data and retroactive corrections, it also requires an understanding of some of its implementation principles.

How we planned to use XTDB

At MarleySpoon, we ship boxes with recipes and ingredients to our customers. The orders and our subscription model are the backbone of the whole bussiness logic and that is also reflected in how we build our software.

The core of the orders system is the orders state machine, and although the states are essentially simple, there might still be cases where we could have discrepancies - and in such cases, we would like to have more options to debug or restore orders to a previous state, as well as retroactively correct their data and push the change to dependent subsystems.

The shift from the legacy monolith architecture to service-oriented architecture coupled with the introduction of new OMS also required us to be more careful when it comes to eventual consistency - the transaction and valid times can be different in the resulting systems so we can’t just work around it by using the persistence stack we’ve used to (e.g. relational DBs with updates in place).

Initially, we considered adding transaction and valid time columns to PostgreSQL to implement bitemporality, as this seemed like a straightforward solution. However, upon further analysis, we realized that this approach would introduce significant complexity to the system design. In particular, any foreign keys would need to take the bitemporal columns into account, meaning that queries would need to consider both the entity relationship and its temporal context. This would require significant changes to the database schema, query design, and application code, and would likely lead to a higher risk of errors and data inconsistencies.

We also considered event sourcing for our needs, but it would add significant incidental complexity, requiring changes to other services’ architectures and a significant amount of application-level code changes to ensure that all events were captured and persisted correctly.

After considering various approaches to implementing bitemporality in our system, we decided to give XTDB a try due to its native support for bitemporality and graph database capabilities. We designed our data model around XTDB’s capabilities and incorporated the database into our Elixir application.

Installing XTDB

There are multiple ways to install and use XTDB:

Use it as a JVM dependency by simply adding it to your JVM project:

        ; deps.edn
        com.xtdb/xtdb-core {:mvn/version "1.21.0"}

Using a pre-built XTDB JAR on a local machine
Through a Docker image

XTDB is developed in the Clojure programming language and it is very convenient to run it from any Clojure program - so we’ve decided to write a simple app in Clojure that would run XTDB for us and provide all required setup.

Installing dependencies

We’ve installed Clojure using asdf version manager as it’s very convenient to pin JVM and Clojure versions:

    # .tool-versions
    openjdk-18
    clojure 1.11.0.1100

The next step was creating a new Clojure app using deps CLI - all of the necessary dependencies were provided in a single file (deps.edn).
XTDB is very modular so we have to install PostgreSQL support, HTTP client and server, metrics, and other tools as separate packages:

    ;; deps.edn
    {:paths ["src"]
     :deps {org.clojure/clojure {:mvn/version "1.11.0"}
            com.xtdb/xtdb-core {:mvn/version "1.21.0"}
            ;; Persistence
            org.postgresql/postgresql {:mvn/version "42.2.18"}
            com.xtdb/xtdb-jdbc {:mvn/version "1.21.0"}
            com.xtdb/xtdb-rocksdb {:mvn/version "1.21.0"}
            ;; HTTP Client
            com.xtdb/xtdb-http-client {:mvn/version "1.21.0"}
            ;; HTTP Server
            com.xtdb/xtdb-http-server {:mvn/version "1.21.0"}}
    }

Application entry point & configuration

One of the primary reasons we developed a wrapper for XTDB was to enable us to run an XTDB cluster in a Kubernetes environment. We wanted to simplify the setup process by allowing configuration through environment variables, rather than relying on external configuration files. This allowed us to easily manage XTDB’s configuration within Kubernetes and provided us with greater flexibility in managing our XTDB cluster.

We’ve created a xtdb.clj file that is the entry point to the database wrapper and which also has all the required configuration there:

    (def db-spec
      {:host (System/getenv "POSTGRES_HOST")
       :port (System/getenv "POSTGRES_PORT")
       :dbname (System/getenv "POSTGRES_DB")
       :user (System/getenv "POSTGRES_USER")
       :password (System/getenv "POSTGRES_PASSWORD")})

    (def config
      {:xtdb.http-server/server {:port 3000
                                 :jwks (System/getenv "XTDB_JWKS")} ; auth
       :xtdb.rocksdb/block-cache {:xtdb/module 'xtdb.rocksdb/->lru-block-cache
                                  :cache-size (* 1024 1024 1024)} ; RocksDB cache size
       :xtdb/index-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                                     :db-dir "/tmp/xtdb/indexes"
                                     :checkpointer checkpoint-config
                                     :block-cache :xtdb.rocksdb/block-cache
                                     :metrics {:xtdb/module 'xtdb.rocksdb.metrics/->metrics}}}
       :xtdb.jdbc/connection-pool {:dialect {:xtdb/module 'xtdb.jdbc.psql/->dialect}
                                   :pool-opts {:maximumPoolSize 10}
                                   :db-spec db-spec}
       :xtdb/tx-log {:xtdb/module 'xtdb.jdbc/->tx-log
                     :connection-pool :xtdb.jdbc/connection-pool}
       :xtdb/document-store {:xtdb/module 'xtdb.jdbc/->document-store
                             :connection-pool :xtdb.jdbc/connection-pool}})

Once the initial configuration is done we can provide a simple entry point function that will start an XT node:

    ;; XT node - hydrated on start
    (def xt-node (atom nil))

    (defn -main
      "Starts a new XTDB node"
      []
      (let [node (xtdb/start-node config)]
        (log/info "Started a new XT node ...")
        (seed/seed node) ; seed data that we need on start
        (xtdb/sync node)
        (log/info "Loaded data into a new XT node")
        (reset! @xt-node node) ; set xt-node with the started node
        node))

    ;; Can be used to run REPL or local XT instance
    ;;
    ;; As running it from CLI can assume passing some command line arguments
    ;; we should accept a list of optional arguments as a function param
    (defn start
      [&args]
      ;; Runs -main function to start a new XT node
      (-main))

Running XT on a local machine

Once our XT app is installed and configured, we can run clj -X xtdb.core/start in order to start it on a local machine.
This will enable web UI and REST API on http://localhost:3000.

Connecting from REPL

To run XT in REPL we can instead just execute clj in shell, given that we are in the root directory of the Clojure project.

That will start a new Clojure REPL, and if we want to start XT from there, it’s sufficient to use functions we’ve implemented beforehand:

    (in-ns 'xtdb.core) ; switches current namespace to XT wrapper's core namespace
    (-main)

Connecting to a remote node

Another important benefit of XT that we unfortunately didn’t explore enough is that by using the http-client dependency, we’re able to connect to any remote node that is accessible to us by HTTP:

    (def remote-node (xt/new-api-client remote-url))
    (xt/submit-tx remote-node [[::xt/put {:xt/id :foo :bar :bar}]]) ; submits a transaction on the remote node

Thus, we can connect from REPL to any production XT instance and run queries / submit transactions there.

REST API

XT is very convenient to use from any Clojure or JVM-based application, however, for clients implemented in other programming languages or different virtual machines, we should be using XT’s REST API.
XT has a very rich HTTP API that covers most of its functionality (although some of it is only available by Clojure/Java client).

API formats

XT supports multiple formats: application/edn, application/json and application/transit+json.

As XT is written in Clojure and it natively supports Clojure’s data types, we were not satisfied with available JSON types and decided to give EDN a try - that way we would have way more supported types:

symbols, e.g. Elixir atoms
decimals
dates and timestamps

However, we had some issues with encoding Elixir/BEAM VM terms to EDN and, in general, the performance of the format - so the Transit/JSON would be an improvement as apart from being compatible with regular JSON and essentially more performant it also has a more precise types conversion.

Main endpoints

GET /_xtdb/status - can be used as a health check for an XT node
GET /_xtdb/entity - gets a single entity from the database
GET /_xtdb/query - performs a single Datalog query
POST /_xtdb/submit-tx - submits database transactions
GET /_xtdb/await-tx - waits until the transaction was indexed on the node
GET /_xtdb/tx-committed - checks if the transaction was successfully committed

Elixir HTTP client

As at that point in time when we started applying XT at MarleySpoon there were no Elixir libraries fully supporting XT’s REST API and covering our needs so we’ve started writing our own HTTP adapter.

Umbrella application

As OMS was the only project where we were trying XTDB, we’ve decided to move all the Clojure code, as well as the new Elixir HTTP into the same umbrella app:

apps/
  ...
  xtdb/

That way we could also easily start an XTDB node right in our project and use it from the Elixir application, e.g. by starting through docker-compose.

HTTP2

XT’s REST API supports the second revision of HTTP format out of the box - which means that we can have a stable and more performant connection between XT clients (application servers) and the database instances.

When running an XTDB cluster in production, HTTP2 can be particularly useful. By using HTTP2, direct connections can be established between the client application and an XTDB node, rather than relying on load balancing across multiple instances. Since XTDB doesn’t enforce an equal state between nodes, the same request could yield different results on different nodes - but using HTTP2 eliminates the issue and ensures consistent results for each request.

However, not every Elixir’s HTTP client supports sending requests using HTTP2 - so we have to search for another option rather than using HTTPoison that we widely use in other projects.
We’ve decided to go with Finch, as apart from supporting HTTP2 it also focuses on performance and provides telemetry support out of the box - which we’ve found very useful for tracing and debugging purposes.

Using HTTP2 with Finch requires some initial configuration as we have to have just one connection in a connection pool:

    # apps/xtdb/lib/application.ex
    children = [
          {
            Finch,
            name: Xtdb.ConnectionPool,
            pools: %{
              default: [count: 1, protocol: :http2, size: 1]
            }
          }
        ]

    Supervisor.start_link(children, strategy: :one_for_one, name: Xtdb.Supervisor)

Telemetry & OpenTelemetry

As was mentioned in a previous section, using Finch as an HTTP Client library is a great step to have a seamless telemetry integration.

Finch provides the next Telemetry events:

request start/stop
request exception
queue start/stop/exception
connection start/stop/exception

and others, but for our purpose, we will be more interested only in the first two events.

In order to integrate the XTDB client with OpenTelemetry we wrote a simple module that watches Finch’s telemetry events and pushes them to the OpenTelemetry collector:

    # Checking for optional opentelemetry dependency
    case Code.ensure_compiled(OpenTelemetry) do
      {:module, _} ->
        defmodule Xtdb.OpenTelemetry do
          @tracer_id __MODULE__

          @spec setup :: :ok
          def setup do
            attach_http_request_start()
            attach_http_request_stop()

            :ok
          end

          defp attach_http_request_start() do
            :telemetry.attach(
              {__MODULE__, :http_request_start},
              [@http_client, :send, :start],
              &__MODULE__.handle_http_request_start/4,
              %{}
            )
          end

          defp attach_http_request_stop() do
            :telemetry.attach(
              {__MODULE__, :http_request_stop},
              [@http_client, :request, :stop],
              &__MODULE__.handle_http_request_stop/4,
              %{}
            )
          end

          def handle_http_request_start(
                _event,
                _measurements,
                %{request: request},
                _config
              ) do
            ...
            Otel.start_telemetry_span(@tracer_id, "XT #{request_url}", %{}, %{
              kind: :internal,
              attributes: attributes
            })
          end

          def handle_http_request_stop(
                _event,
                _measurements,
                %{request: _request, name: ConnectionPool, result: {_, %{status: status} = result}},
                _config
              ) do
            context = Otel.set_current_telemetry_span(@tracer_id, %{})
            Span.set_attribute(context, :"http.status", status)
            ...
            OpenTelemetry.end_telemetry_span(@tracer_id, %{})
          end
        end

      _ ->
        nil
    end

Encoding EDN

In Elixir are using Eden library in order to decode and encode data from and to EDN format. In most cases it works without any issues, however, for decimals we had to implement protocol support for Elixir Decimal type:

    # Implements Eden.Encode protocol for Decimal structs
    defimpl Eden.Encode, for: Decimal do
      # Adds a decimal digit number so it can be picked by EDN
      @spec encode(Decimal.t()) :: String.t()
      def encode(decimal), do: decimal |> Decimal.to_string() |> Kernel.<>("M")
    end

That way you can also implement support of any other custom type or struct that you might need to persist in XT.

Testing XTDB

Because XTDB is an immutable database, it’s not so simple to delete data from it. This can have implications when it comes to testing, as traditional methods of clearing and resetting a database may not be effective. To integrate XT into a test suite, the most straightforward approach is to run an in-memory node alongside the suite. This allows for more granular control over data, as the in-memory node can be reset or recreated as needed.

We’ve achieved that by using the in-memory XTDB Docker image in our docker-compose setup:

    # Wait for Docker container to be ready
    wait:
      image: dokku/wait

    xtdb_test:
      image: juxt/xtdb-in-memory:1.21.0
      ports:
        - 3000:3000

and also created a simple shell script that we use in order to run the integration test suite:


    docker-compose up --remove-orphans -d xtdb_test
    docker-compose run wait -c xtdb_test:3000
    docker-compose run mix_test
    docker-compose stop xtdb_test mix_test

In the case of unit tests, as any request coming through the XTDB client is basically an HTTP request, we could also use VCR cassettes as mocks in order to avoid sending real requests to the test instances.

What’s next?

In this article, we’ve discussed how XTDB can be used alongside applications written in Elixir and demonstrated how to implement a simple HTTP client for working with the database. We also covered how to develop and test applications using XTDB, and the importance of running an in-memory node for testing purposes.

In the third and final part of this series, we’ll be sharing how we used Docker and docker-compose to set up a local development environment for XTDB, as well as how we deployed and ran it in production. We’ll also be discussing caveats and issues we encountered, and how we addressed them.

We can also recommend some more reading on the topic if you’re interested in developing with XTDB:

Happy Hacking and stay tuned!

Thanks Carsten for the review!
Cover image credit: DALL·E 2.

Constant Doubts

Carsten Zimmermann — Thu, 23 Mar 2023 12:58:47 +0000

I always had mixed feelings towards Ruby constants. First, they're not that constant to begin with. You can reassign a constant at runtime freely and all you get is a warning:

irb(main):001:0> FOO = :bar
irb(main):002:0> FOO = :lol
(irb):2: warning: already initialized constant FOO
(irb):1: warning: previous definition of FOO was here

Contrary to popular belief, a #freeze won't help, either. Yes, it makes the object you're assigning immutable, but it doesn't prevent another constant assignment:

irb(main):001:0> FOO = "bar".freeze
irb(main):002:0> FOO = "baz".freeze
(irb):2: warning: already initialized constant FOO
(irb):1: warning: previous definition of FOO was here
irb(main):003:0> # 🤷

(It would be nice to be able to change warning into an error, but there's certainly no RUBYOPT flag that I know of.)

But most importantly: unless they're defined in the top level object space, I consider them implementation details. If an instance of A::Nested::Class accesses Some::Other::CONST, it crosses 5-6 boundaries (depending on where you start counting). That's not actually respecting that module's privacy and also a violation of the Law of Demeter.

`private_constant` all the things!

By default, constants are public. Most software engineers are very eager to work with the private keyword to limit the public API of their instances, but it's rarer to see that same rigor applied to class or module-level constants.

Ruby has private_constant since basically forever (MRI v1.9.3 to be precise). It accepts one or more symbols referring to defined constants in its scope:

module MyModule
  FOO = "bar".freeze
  LOL = :wat?

  private_constant :FOO, :LOL

  def self.foo
    FOO
  end
end

Any attempt to access MyModule::FOO from outside MyModule will raise a NameError ("private constant MyModule::FOO referenced").

Please note that MyModule.foo still works (and returns the frozen string "bar") as it only accesses the private constant from within its defining scope.

(Singleton) Methods Over Constants

I mentioned I consider constants implementation details. A named identifier for a magic value, maybe something configurable and set at load time. And sometimes, you want to expose that to other components in your applications.

As shown in the code snippet above, you can always create a singleton method / module function that wraps around a private constant.

In my opinion, methods are in all cases superior to CONSTANTS:

you can defer (lazy load) the assignment
you can memoize (@class_var ||= ...) what's being assigned
you can delegate the method call
it's easier to stub a method than a constant in your tests
it pairs well with making class/module-level value configurable on the application level, e.g. using a well-known MyModule.configure(&block) format or by using dry-config
it feels much more OOP to send a message to an object than working with its constants (also, I always feel that CONSTANTS YELL AT ME in the source code)

Points 1..5 all give you a great forward compatible way to refactor how your magic value is used, all for the small price of making your constants private and adding a getter singleton method around it if you really need to expose it to the outside world right away.

Enforce Explicit Constant Visibility

Unfortunately, there is no way to make all constant private by default, but RuboCop includes a RuboCop::Cop::Style::ConstantVisibility cop to at least make the constant scope explicit.

I like that it makes you stop and think about what you're doing.

Named Classes Are Constants

Now, constants aren't only referred to by UPPERCASE identifiers: all class and module names are in fact constants, so you could argue that my criticism about accessing Some::CONSTANT directly must also extend to a form like Nested::Class.new.

And indeed it does, but that will be the topic of the next article. 😀

Cover image credit: DALL·E 2.

Bitemporality, or how to change the past

Alex Kuznetsov — Fri, 06 Jan 2023 08:34:00 +0000

We can definitely see the whole history of humanity as a chain of events:
tiny events, big events, huge events, crucial events – some of them were negligible and some, on the contrary, predefined the next link in the chain and changed the way our world looks now.

However, the more details we dig out about our history, the more white spots we erase, the more surprising and even contradictious things we see – things that sometimes can completely refute our views and assumptions. Things that we thought and believed to be facts can be invalidated – and we will need to alter the past.

Every kind of record system is not perfect and can contain errors. But what if we wanted to build a system that is resilient to such errors? What if we wanted to change the past but keep the history as a track of facts?

Updates in place

Throughout almost all the existence of computers and computer science, resources were the biggest limitation and one of the most complex issues.

Our progress allowed us to switch from bytes to kilobytes, from kilobytes to mega and gigabytes. Modern PCs, smartphones, and cloud systems can easily handle virtually unlimited amounts of data.

The storage, which was physically enormous at the beginning of the computer era and started with devices taking over the halls and rooms, moved to CDs, floppy disks and hard drives – so we could even carry movies, tons of photos and music in our pockets. The storage, which always seemed to be the problem, became the smallest of our headaches when it came to building software.

Nonetheless, the habit (the bad one) we inherited from the dawn of the computer era is still there and arguably is one of the biggest developer’s headaches – mutability. Before the progress in the storage capacity, we were very careful with how we consumed the RAM and disk space, and that forced us to mutate variables and data instead of persisting them in the immutable way, i.e. saving a new record instead of updating an existing one.

That habit is still haunting us, even though functional programming languages and the immutability approach are on the rise, we still (mostly) update our records in place – as that is the way most popular databases are built.

An audit system

How can we design a record tracking system using a common mutable DB? Let’s say we use PostgreSQL and define a simple audits table:

Company	Issues found	Auditor	Date
BullSheepInc	180	Joe	15.09.2021
OverHypeD	420	Max	14.02.2022
BullSheepInc	0	Max	15.02.2022

However, one and a half months ago we’ve discovered that Max forgot to write down the records for BullSheepInc — and as the column has default of 0, we were unaware of the change for quite some time. Now, if we want to fix it, we have to overwrite the existing record:

Company	Issues found	Auditor	Date
BullSheepInc	180	Joe	15.09.2021
OverHypeD	420	Max	14.02.2022
BullSheepInc	101	Max	01.04.2022

The drawbacks of such an approach are clearly visible: we’ve lost track of the changes and forgot about the mistake. As our reports were already sent, there won’t be any punishment for BullSheepInc.

Temporality

The next step from mutability to our goal of designing a perfect track record system will be to persist changes as facts, instead of updating data in place.
In order to build a truly immutable system we want to disallow overriding records in our audit system. Instead, we will be storing any kind of data change as a
separate row in the database:

Company	Issues found	Auditor	Date
BullSheepInc	180	Joe	15.09.2021
OverHypeD	420	Max	14.02.2022
BullSheepInc	0	Max	15.02.2022
BullSheepInc	101	Max	01.04.2022

It looks like this option worked out better for us; now we see that the inventory record change was tracked and we’ve actually discovered some issues.
We could stop there and pretend we’ve built the most advanced audit system, but that would be too far away from being the truth, as soon we’ve got yet another request …

Retroactive changes

Once our employees started seeing not just audit records but also data fixes, John has recalled that it was actually him who did the counting, and the amount of found issues was actually 99 instead of 101.
We’ve got a serious problem now as the new record doesn’t fit into our data model:

Company	Issues found	Auditor	Date
BullSheepInc	180	Joe	15.09.2021
OverHypeD	420	Max	14.02.2022
BullSheepInc	0	Max	15.02.2022
BullSheepInc	101	Max	01.04.2022
BullSheepInc	99	John	01.05.2022

Which record is really valid now? Should we trust Max or John? How should we define what was the error and how it was corrected ?
That’s where the concept of bitemporality comes to the rescue.

Bitemporality

In the example above, we have only one time column: the record, or transaction date.
Bitemporality assumes adding another time dimension — the so-called valid time or effective time — along the transaction time for tracking when the change really happened.

Transaction time represents the time when the record was inserted into the data storage. This can be quite useful for audit purposes, tracking changes and event sourcing.
Valid time represents when the change became valid and happened in the real world.

If we follow these definitions we can say that transaction time is the time we thought the data was correct at that point in time — and it was actually correct on the valid time:

On the 15th of February, we’ve thought Max has not found any issues.
On the 1st of April, Max corrected the number of issues to be 101.
On May 1st, we’ve discovered that John actually found 99 issues.
In reality, we want the actual amount of issues recorded to be 99 as of 15th of February.

In a bitemporal system transaction time is immutable and can be only increased while valid time can be any past or future timestamp.
Let’s see how we can redesign the audit system using these two time dimensions:

The perfect audit system™

Now that we know how to utilise transaction and valid dates, we can change our records by writing the record time as transaction date and
time when it became valid as valid time:

Company	Issues found	Auditor	Transaction date	Valid date
BullSheepInc	180	Joe	15.09.2021	15.09.2021
OverHypeD	420	Max	14.02.2022	14.02.2022
BullSheepInc	0	Max	15.02.2022	15.02.2022
BullSheepInc	101	Max	01.04.2022	15.02.2022
BullSheepInc	99	John	01.05.2022	15.02.2022

Let’s execute some queries to our database:

    module Audit
      # @return [Hash] a hash with auditor, issues found and transaction date fields
      def get_record(company, valid_date = nil, transaction_date = nil)
        # calling the DB ...
      end
    end

    > Audit.get_record("BullSheepInc")
    # {auditor: "John", issues_found: 99, transaction_date: "01.05.2022"}
    > Audit.get_record("BullSheepInc", "15.09.2021")
    # {auditor: "Joe", issues_found: 180, transaction_date: "15.09.2021"}
    > Audit.get_record("BullSheepInc", "01.01.2021")
    # nil - we didn't inspect the company as of 01.01.2021
    > Audit.get_record("BullSheepInc", "15.02.2022", "01.04.2022")
    # {auditor: "Max", issues_found: 101, transaction_date: "01.04.2022"}

As you can see, now we have the latest correct value returned by default,
but we can also fetch the record the record as it was on a given valid date in the past.

Use-cases

Bitemporality can be proven useful for any system where you have a track of the data and where it’s possible to have errors and recover them:

payrolls, payment systems
auditing
risk systems
blockchains
insurance
compliance & privacy
temporal data management
event-based systems
distributed transactions

Cross-time Database

Supporting bitemporality in an existing database might be not a trivial task, especially when it comes to the traditional relational database where we all relations between tables should also take into account bitemporal columns.

At the moment, the most prominent open-source solution is XTDB (or cross-time) database developed by JUXT which has a lot of benefits compared to its competitors:

Bitemporal at its core
Supports retroactive corrections
Document&graph based (ultimately a store of versioned documents)
Datalog queries and SQL support
Data eviction (supports eviction of active and historical data to assist with technical compliance for information privacy regulations)
Distributed and scalable
Unbundled database (can be deployed on top of many existing technologies and databases like Kafka, JDBC, AWS S3)
Can be easily integrated into any existing JVM application or connected using its REST API

As we will explore more in the next articles, XTDB can be used as a ready solution for building immutable and bitemporal software.

What’s next?

As one can see, bitemporality can be a perfect match for cases where we build systems that are:

Tracking the history of changes or how data was changing over time
Can potentially have errors, corrections or data adjustments

If we take that concept as the cornerstone of such systems there will be way more chances they will be successful and
we will escape from mutability issues.

We can also recommend some more reading on the topic:

In the upcoming article we will share our experience working with XTDB, connecting to it from an Elixir application and what we learned from it.

Happy Hacking and stay tuned!

Thanks Carsten for the review!

What's the point? The 3 Es.

Carsten Zimmermann — Tue, 05 Apr 2022 12:00:54 +0000

Musings on the arbitrary number we attach to tasks & stories

Once per moon, we find ourselves discussing both what dimensions we should consider when we’re doing our Backlog Refinement and what types of work should receive Story Points to begin with. Occasionally, the term →“Business Value” is tossed into the conversation, but for my heavily backend-focussed team, it is very difficult to attach that particular metric to a highly technical task – and we currently have a lot of them.

This article originally aimed to conclude that conversation, a publicised Working Agreement if you will. It may serve as inspiration to others who wish to look at their pointing strategy through the lens of a Backend team. But as with most things on the internet, YMMV.

Issue Types

We work with Product Backlog Items (PBI) that can be categorised as one of the following:

Story

Captures work from an end user’s perspective, with varying definitions of what an end user is (for us, it can be another system consuming our public API). They typically follow the format of “As <who> <when> <where>, I want <what> because <why>”.

Stories are pointed.

Task

Describes an often technical aspect of an overarching goal. This could be a necessary refactoring, foundational work, paying back technical debt, or general housekeeping chores.

We point Tasks.

Bug

Represents a defect, something that isn’t working as intended. Bugs have varying priorities: some have a very low impact with known workarounds, the more urgent ones affect production and need to be addressed right away.

Some Bugs we point, some Bugs we don’t:

High-priority bugs that need to be added to current Sprint probably represent rework – a missed edge-case or something that slipped through QA. As such, it should not count towards the team’s →Velocity as that work should have gone into previous Sprints… so to speak
Bugs for which we have workarounds or whose impact is considered low should be part of the normal prioritisation process. Attaching a number to it will help weighing against other items in the Backlog.

Spike

Work to explore a new technology, test new processes, eliminate unknowns (see below, →Entropy). The goal is not to ship production-ready code, but rather to provide a proof of concept. Code produced while working on a Spike might deliberately forgo tests or sad path handling.

The main output of a Spike is information. The information is shared with the team. We currently do not point Spikes, but we time-box them (~2 days).

Epic

A container for related issues of various types. Ideally, they have a defined end-date and all its member issues (Stories & Tasks, mainly) are defined & scoped together to avoid too much context switching.

Possible anti-pattern: using an Epic to group related tickets in such a way that the Epic grows and grows.

Interpreting Points: The Three Es (3Es)

I understand story points to be one of the following:

Entanglement (as in: how many connections / interactions / dependencies)
Effort (as in: how time consuming is the actual work)
Entropy (as in: state of disorder, The Unknown)

Entanglement

Entanglement reflects how many moving parts have to be considered, the literal complexity of the feature. Higher complexity will translate to more diligence considering side-effects or edge-cases, as well as requiring a more thorough QA process.

Effort

Effort can be a measurement for how laborious a task is, even if it’s not rocket science. An example would be restructuring a code base where the goal is clear, but it will involve renaming many files and module names.

Entropy

Entropy expresses the uncertainty around a given task. It can cover blind-spots about the implications of a change, lack of experience with the domain or technology, a placeholder for the unknown Unknowns that might be uncovered while working on the task.

Note: if there’s too much entropy, creating a →Spike first is advisable.

Business Value

Now, the Three Es don’t include the hallowed Business Value. After all, everything in agile software development is about continuously delivering value, right?

Unfortunately, there is no objective scale by which we can measure Business Value. For some cases, “revenue added / cost saved” would be a good unit of measure, but “customer satisfaction” is nigh impossible to translate to money, whereas “better fault tolerance” requires a lot of effort (and guesswork) to put a € sign next to it.

Business Value is also time-dependent: it exists within the context of the organisation’s current situation and direction. Whatever arbitrary value we assign as Business Value for a story today might receive a completely different rating next month when re-evaluated.

Business Value is not a metric to feed into a team’s Velocity. It’s a means to prioritise work!

Further reading: Business Value by Erik de Bos.

Velocity & Capacity

In Agile software development processes, the Sprint Velocity is a metric for the output of a team, measured in Story Points completed. Cynically, one could argue that it’s also a metric on how good the team is at estimating, but let’s assume that outliers in estimation are inevitable and are evened out over time.

A (somewhat) deterministic pointing system where units of work (represented in PBIs) can be compared is key to assess the output of a team. Hopefully, the 3Es above help with that.

Now, Velocity as a metric for a team’s output (i.e. the sum of completed story points) is meaningless if it’s not correlated to the input of the Sprint: the person-days that were available. It’s the Story Points / Available Capacity that really matters. The rationale being: one Sprint might have lower output because ⅓ of the team was recharging their batteries on well-deserved PTO (reduced input).

If you want to track the relative efficiency of your Sprint, the relation of completed Story Points to Days Available provides the best insight and it also leads to more precise capacity planning: by calculating, say, a 4-Sprint-average of that ratio, you’ll end up with a multiplier to give you a good idea about how many Story Points you might complete in the next Sprint.

Sprint	1	2	3	4
A: Points completed	20	22	28	18
B: Available Days	40	35	42	22
Capacity Modified (A/B)	0.5	0.63	0.67	0.82

The 4-Sprint-average is (0.5 + 0.63 + 0.67 + 0.82) / 4 = 0.655. If you have 36 person-days available in your upcoming Sprint, you should aim for ~24 Story Points.

It’s controversial if you want to count unfinished business towards your delivered output. I get why you wouldn’t want that – a User Story that isn’t fully done doesn’t provide value – but we also established that we won’t look at value.

Being able to accurately plan what can be delivered in a Sprint is much more tangible than the incorporeal value and to that end, we split issues based on what was completed and what will roll over to the next Sprint.

Conclusion

In general, we already have a good grasp on how how many points to attach to a PBI – being primarily an #Elixir team, we have mentally assigned @default_points to 2 and just use the ~~module~~ brain attribute in Scrum Poker. This is a good sign, as we're apparently doing ok at splitting work into bite-sized chunks already. But for more complex issues and when the estimates given differ a lot, it's good to do a sanity check by looking at the three dimensions Entanglement, Effort, and Entropy separately.

Additionally, finally having resolved that considering Business Value as something that should go into a PBI's story point estimation was a huge win.

Header photo credit: PxHere, CC0 Public Domain

Using VCR to Mock Your Requests

Ana Schwendler — Fri, 18 Jun 2021 10:05:31 +0000

In the past months, I've been working with a lot of external API requests. Since I'm part of the Marley Spoon’s logistics team, our challenge is to make sure our requests are done in time and everything is delivered in good shape.

Part of this task, as a software developer, is making sure the code I write is well tested, and does what it is supposed to do. Our software has many integrations with our logistics partners. When integrating with their systems, we don't want to hit external APIs with real requests every time we run our tests. This not only generates unnecessary requests that can cause problems with API rate limits and other unwanted side-effects. It also makes them run slower. To avoid that, we pretend to make real requests and there is a tool that helps "faking" these calls: by recording real data from a real network request, we are able to stub a third party’s web service response. The saved data is called a "cassette tape" and the name of the tool I refer to is VCR.

In this post I'll explain how I've been learning to use it and why I think it is useful.

What is a mock test?

Since this is not about the mocking technique, I'll just briefly introduce it and link to a useful article on the topic. In this post we can see mock testing defined as:

Mocking means creating a fake version of an external or internal service that can stand in for the real one, helping your tests run more quickly and more reliably. When your implementation interacts with an object’s properties, rather than its function or behavior, a mock can be used.

… and that is exactly the reason why we use VCR.

What is VCR?

According to their documentation:

Record your test suite's HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests.

Which means that when you first run your test with the VCR syntax, it will record what happened and next time you run the test again, you will have the recorded version ready to stand in for the real request.

How to use VCR?

I have a basic example here.

So basically, whenever you want to test a part of code that requires an external web request, you use the .use_cassette method to state that you want VCR to deal with that with a cassette file. If there already is data with a pre-recorded HTTP interaction, VCR will use it. If not, VCR will automatically create a cassette file (this time making a real request) based on the request made in the test.

But still, if you are looking forward to making real requests, VCR also offers different record modes, which can be checked here

Example:

require 'rubygems'
require 'test/unit'
require 'vcr'

VCR.configure do |config|
  config.cassette_library_dir = "fixtures/vcr_cassettes"
  config.hook_into :webmock
end

class VCRTest < Test::Unit::TestCase
  def test_example_dot_com
    VCR.use_cassette("synopsis") do
      response = Net::HTTP.get_response(URI('http://www.iana.org/domains/reserved'))
      assert_match /Example domains/, response.body
    end
  end
end

Here you are requesting to use a cassette called synopsis, which mocks a request to iana.org.

If this is running the first time, VCR will generate a "cassette" file, which will be stored at fixtures/vcr_cassettes, which for the example will be called synopsis.yml.

This is what fixtures/vcr_cassettes/synopsis.yml looks like (I've removed some parts as it was too big):

--------
http_interactions:
- request:
    method: get
    uri: http://www.iana.org/domains/reserved
    body:
      encoding: US-ASCII
      string: ''
    headers:
      Accept-Encoding:
      - gzip;q=1.0,deflate;q=0.6,identity;q=0.3
      Accept:
      - "*/*"
      User-Agent:
      - Ruby
  response:
    status:
      code: 200
      message: OK
    headers:
      Date:
      - Fri, 18 Jun 2021 08:06:40 GMT
      Server:
      - Apache
      Vary:
      - Accept-Encoding
      Last-Modified:
      - Thu, 21 May 2020 22:41:39 GMT
      X-Frame-Options:
      - SAMEORIGIN
      Expires:
      - Fri, 18 Jun 2021 09:56:45 GMT
      Referrer-Policy:
      - origin-when-cross-origin
      X-Content-Type-Options:
      - nosniff
      Age:
      - '595'
      Content-Type:
      - text/html; charset=UTF-8
      Cache-Control:
      - public, max-age=21603
      Content-Security-Policy: /*cropped security policy*/
      Transfer-Encoding:
      - chunked
    body:
      encoding: ASCII-8BIT
      string: !binary |-
        /*cropped binary string*/
    http_version:
  recorded_at: Fri, 18 Jun 2021 08:06:40 GMT
recorded_with: VCR 5.0.0

From now on your test suite will use this cassette file. It will also run faster, as it has the file in place and doesn't need to make the request which can take more time.

Examples in real life

One example of project that uses VCR is this site-search-ruby client from elastic: https://github.com/elastic/site-search-ruby

In their context of Search, when searching all DocumentTypes in the engine, they use a cassette called engine_search to mock a request:

https://github.com/elastic/site-search-ruby/blob/master/spec/client_spec.rb

This is the cassette file that is used in this call: https://github.com/elastic/site-search-ruby/blob/master/spec/fixtures/vcr/engine_search.yml

Conclusion

VCR seems to be a very reliable tool to fake requests to APIs. But we still need to be aware that sometimes APIs can change and with that we need to record our tests again. This process should be simple and easy to reproduce, so for more tips regarding VCR use, I recommend this blog post.

Hiding confidential credentials in vcr_cassettes

There is a configuration option available to filter sensitive data that can be used to prevent it from being written to the cassette files, the documentation is here. This is important, as you want to track your cassette YML files in your source control management tool. And secrets don’t belong there.

References

https://github.com/vcr/vcr

https://github.com/elastic/site-search-ruby

https://circleci.com/blog/how-to-test-software-part-i-mocking-stubbing-and-contract-testing/

https://fabioperrella.github.io/10_tips_to_help_using_the_VCR_gem_in_your_ruby_test_suite.html

Thanks Memuna for reviewing it! 🎉

DEV Community: Marley Spoon

The Four Principles of Design Thinking in Software Architecture

Thinking in Abstractions

Use Case

First Try: IF-ELSE Statement

Pros

Cons

Second try: Abstraction Through Inheritance

Pros

Cons

Conclusion

Retro Guide

Introduction

Directives

Context

Start or End of a Project

Team Changes

Implementation of New Processes

Development Cycle

External Events

Performance Metrics

Customer Feedback

Organizational Objectives

Evaluation of Training or Culture Changes

Focus on the Current Sprint

Specific Goal Orientation

Connection to Long-Term Goals

Identification of Trends Over Time

Participant Preparation

Standout moments in the sprint.

Effective practices or processes.

Exceptional collaboration between team members.

Individual skills that contributed to success.

Aspects of the work that led to overall success.

Positive changes compared to the previous sprint.

Each team member's main contributions.

Main obstacles or challenges faced.

Communication difficulties negatively impact the team.

Aspects of the work process that can be optimized.

Recurring issues that need addressing.

Gaps in the team’s skills or knowledge.

Handling external pressures or tight deadlines.

Tasks or activities that did not contribute significantly.

Checkin

Continuous Improvement

Filtering

Checkout

Conclusion

References

The Request and Response Model in the Clean Architecture

Introduction

The Request and Response Model

Request

Response

Implementing

Benefits of the Request and Response Model

References

Understanding Bounded Contexts in Ruby

Introduction

Understanding Bounded Contexts

Consuming and Wrapping Plan Objects

Conclusion

References

Taming Time: how to run XTDB in production

XTDB 1

Deploying XTDB

Containerised XTDB

Uberjar

Dockerfile

Graceful shutdown

Authentication

Kubernetes

Running XT in production

JDBC

Kafka

Checkpoints

AWS S3

Caveats

Memory consumption

RocksDB tuning

First Try: `IF-ELSE` Statement

`private_constant` all the things!