DEV Community: Mark Andreev

FIX Protocol System-Level Implementation: Session Management Deep Dive

Mark Andreev — Mon, 13 Oct 2025 19:26:24 +0000

The Financial Information eXchange (FIX) protocol serves as the standardized messaging framework for electronic trading across global financial institutions. The session-level protocol remains consistent across implementations, providing the foundation for reliable communication that processes billions of messages daily with sub-millisecond latency requirements.

This article examines the system-level implementation of FIX protocol, focusing on four fundamental session management messages: Logon, Logout, Heartbeat, and Resend Request. These components form the infrastructure layer that every FIX implementation must handle correctly, regardless of the business logic built on top.

Note: Application-level business messages (orders, executions, market data) are excluded to maintain focus on the foundational session layer.

FIX Message Structure and Encoding

Tag-Value Format Specifications

FIX protocol employs tag-value encoding where each field appears as tag=value pairs separated by the Start of Heading (SOH) character (ASCII 0x01). The SOH delimiter is non-printable, requiring careful handling during debugging and logging.

Every FIX message adheres to this structure:

8=FIX.4.4|9=89|35=A|49=SENDER|56=TARGET|34=1|52=20251012-14:30:15|108=30|98=0|141=Y|10=178|

Note: Pipe characters (|) represent SOH (0x01) for readability throughout this article.

(View in FixParser)

Message Anatomy and Mandatory Fields

Three fields maintain strict positional requirements:

Tag 8 (BeginString): Protocol version identifier, always first field
Tag 9 (BodyLength): Byte count starting immediately after tag 9's delimiter through the byte preceding the checksum field's tag, excluding BeginString and BodyLength themselves but including all SOH delimiters within that range
Tag 35 (MsgType): Message type identifier, always third field
Tag 10 (CheckSum): Three-digit checksum value, always last field

The BodyLength calculation excludes tags 8, 9, and 10 along with their delimiters, but counts every byte in between, including field delimiters and the values themselves.

Checksum Algorithm Implementation

The CheckSum field (tag 10) ensures message integrity through a modulo 256 algorithm. The calculation process:

Sum all bytes from BeginString (tag 8) through the delimiter preceding the CheckSum field
Apply modulo 256 to the sum
Format the result as a zero-padded three-digit string

For example, if the byte sum equals 274, the calculation yields 274 % 256 = 18, transmitted as "018". This lightweight validation catches transmission errors without cryptographic overhead.

Message Validation Sequence

Implementations should validate messages in this specific order to prevent expensive processing of malformed data:

BeginString format verification
BodyLength calculation and boundary check
MsgType field presence and validity
Required field presence based on message type
CheckSum calculation and comparison
Sequence number verification against expected value

This ordering ensures that structurally invalid messages are rejected before performing computationally intensive validation steps.

Session vs Connection Lifecycle

A critical distinction exists between FIX sessions and connections. A FIX session represents a logical relationship between two parties that spans multiple sequential (not concurrent) physical connections, maintaining continuous sequence numbering across disconnects and reconnects.

A single connection consists of:

TCP establishment
Logon message exchange
Business message flow
Logout and TCP closure

A session can span an entire trading day with multiple connection attempts, preserving sequence state across each reconnection. The sequence number series begins at 1 and continues monotonically throughout the session's lifetime, potentially across dozens of individual connections.

Session vs Application Message Layers

FIX operates at two distinct protocol layers:

Session Level Messages:

Manage connection lifecycle and reliability
Enforce ordered message delivery
Handle sequence gaps and recovery
Types: Logon (A), Logout (5), Heartbeat (0), Test Request (1), Resend Request (2), Sequence Reset (4), Reject (3)

Application Level Messages:

Transport business-specific trading data
Examples: New Order Single (D), Execution Report (8), Market Data Request (V)

Session Management Messages

Logon Message (MsgType=A)

The Logon message initiates a FIX session and must be the first message transmitted by the connection initiator. Authentication parameters and session configuration are established during this exchange.

Required and Optional Fields

Tag 35: MsgType = 'A'
Tag 98: EncryptMethod (0 = none, 1 = PKCS, 2 = DES, 3 = PKCS/DES)
Tag 108: HeartBtInt (heartbeat interval in seconds)
Tag 141: ResetSeqNumFlag (Y/N - controls sequence number initialization)
Tag 554: Password (authentication credential)
Tag 553: Username (authentication credential)

Note that tag 98 (EncryptMethod) refers to FIX application-layer encryption. Most exchanges require TLS/SSL at the transport layer even when EncryptMethod=0, as transport and FIX-layer encryption operate independently.

Logon Message Example

Initiator to Acceptor:

8=FIX.4.4|9=89|35=A|49=TRADER_SYSTEM|56=EXCHANGE_GATEWAY|34=1|52=20251012-14:30:15.123|98=0|108=30|141=Y|553=trader123|554=secretpass|10=178|

(View in FixParser)

Field Interpretation:

8=FIX.4.4: Protocol version
9=89: Body length in bytes
35=A: Logon message identifier
49=TRADER_SYSTEM: SenderCompID
56=EXCHANGE_GATEWAY: TargetCompID
34=1: Message sequence number
52=20251012-14:30:15.123: UTC timestamp
98=0: No FIX-layer encryption
108=30: 30-second heartbeat interval
141=Y: Reset sequences to 1
553=trader123: Username
554=secretpass: Password
10=178: Checksum

Acceptor Response (Successful):

8=FIX.4.4|9=78|35=A|49=EXCHANGE_GATEWAY|56=TRADER_SYSTEM|34=1|52=20251012-14:30:15.456|98=0|108=30|141=Y|10=156|

(View in FixParser)

Acceptor Response (Authentication Failed):

8=FIX.4.4|9=87|35=5|49=EXCHANGE_GATEWAY|56=TRADER_SYSTEM|34=1|52=20251012-14:30:15.789|58=Authentication failed|10=234|

(View in FixParser)

Authentication Failure Handling

When authentication fails, the acceptor must send a Logout message rather than simply closing the connection. The Logout should include tag 58 (Text) with the specific failure reason. The acceptor should wait a configurable period (typically 1-2 seconds) before closing the socket to ensure complete transmission of the Logout message.

Sequence Number Initialization

The ResetSeqNumFlag (tag 141) determines sequence number behavior. When set to 'Y', both parties reset their outbound and expected inbound sequence numbers to 1. When set to 'N', sequence numbers continue from the previous session's final values, requiring both parties to maintain persistent state across connection failures.

Logout Message (MsgType=5)

The Logout message provides graceful session termination. Either party may initiate logout, and proper implementations require bidirectional Logout exchange before TCP closure.

Message Structure

Tag 35: MsgType = '5'
Tag 58: Text (reason for logout)

Logout Scenarios

Normal Shutdown:

8=FIX.4.4|9=73|35=5|49=TRADER_SYSTEM|56=EXCHANGE_GATEWAY|34=157|52=20251012-16:00:00.000|58=End of trading day|10=145|

Response Logout:

8=FIX.4.4|9=65|35=5|49=EXCHANGE_GATEWAY|56=TRADER_SYSTEM|34=298|52=20251012-16:00:00.123|58=Goodbye|10=123|

Logout Due to Sequence Error:

8=FIX.4.4|9=89|35=5|49=EXCHANGE_GATEWAY|56=TRADER_SYSTEM|34=299|52=20251012-15:45:30.567|58=Sequence number too low: expected 158, received 150|10=187|

Graceful Termination Requirements

The logout initiator must wait for the counter-party's acknowledgment Logout message before closing the socket. However, implementations must include a timeout mechanism (typically 2-5 seconds) to prevent indefinite waiting if the counter-party fails to respond. After timeout expiration, the socket should be forcibly closed to prevent resource leaks.

Heartbeat Message (MsgType=0)

Heartbeat messages maintain connection liveness during idle periods. They are transmitted automatically when no other messages have been sent within the agreed heartbeat interval.

Message Components

Tag 35: MsgType = '0'
Tag 112: TestReqID (present only when responding to Test Request)

Heartbeat Examples

Spontaneous Heartbeat:

8=FIX.4.4|9=58|35=0|49=TRADER_SYSTEM|56=EXCHANGE_GATEWAY|34=89|52=20251012-15:30:45.000|10=123|

Response to Test Request:

8=FIX.4.4|9=75|35=0|49=EXCHANGE_GATEWAY|56=TRADER_SYSTEM|34=234|52=20251012-15:30:50.000|112=TEST_ID_001|10=189|

Heartbeat Timer Behavior

The heartbeat timer resets after transmitting any message type, not exclusively heartbeats. This prevents unnecessary heartbeat traffic during active trading periods. When responding to a Test Request (MsgType=1), the Heartbeat must echo back the TestReqID value received in tag 112.

If a party fails to receive any message within the heartbeat interval plus a reasonable transmission buffer, it should issue a Test Request. Continued silence after the Test Request indicates connection failure and typically triggers session termination.

Resend Request Message (MsgType=2)

The Resend Request message implements gap recovery by requesting retransmission of missing messages. This mechanism ensures reliable, ordered delivery despite network packet loss or out-of-order TCP segment arrival.

Request Structure

Tag 35: MsgType = '2'
Tag 7: BeginSeqNo (first sequence number to resend)
Tag 16: EndSeqNo (last sequence number to resend, 0 = infinity)

Resend Request Examples

Request Specific Range:

8=FIX.4.4|9=58|35=2|49=TRADER_SYSTEM|56=EXCHANGE_GATEWAY|34=45|52=20251012-14:45:15.000|7=35|16=42|10=167|

Request Through Current Message:

8=FIX.4.4|9=57|35=2|49=EXCHANGE_GATEWAY|56=TRADER_SYSTEM|34=123|52=20251012-14:45:20.000|7=98|16=0|10=145|

The infinite EndSeqNo scenario (tag 16=0) requests all messages from BeginSeqNo through the most recently sent message. This simplifies gap recovery when the requester is uncertain about the actual gap size.

PossDupFlag and OrigSendingTime

Resent messages must include PossDupFlag (tag 43) set to 'Y' and OrigSendingTime (tag 122) containing the timestamp from the original transmission. This allows receivers to detect and handle duplicate processing scenarios:

8=FIX.4.4|9=XXX|35=D|49=BROKER01|56=CLIENT01|34=23|52=20251012-09:16:30.000|43=Y|122=20251012-09:10:15.000|...|10=XXX|

The combination of sequence number, PossDupFlag, and OrigSendingTime enables the receiver to determine whether a message represents a genuine retransmission or a critical protocol violation.

Sequence Reset with GapFillFlag

When requested messages contain only administrative messages that don't require retransmission, the sender responds with a Sequence Reset (MsgType=4) containing GapFillFlag (tag 123) set to 'Y':

8=FIX.4.4|9=78|35=4|49=EXCHANGE_GATEWAY|56=TRADER_SYSTEM|34=35|52=20251012-14:45:25.000|123=Y|36=43|10=234|

The NewSeqNo field (tag 36) is set to one greater than the last administrative message sequence number in the gap. This informs the requester that no application-level messages need recovery for that range.

Nested Gap Recovery

If a Resend Request itself has a sequence gap (the Resend Request's sequence number is higher than expected), the receiver faces nested gap recovery. The receiver must prioritize the new gap by issuing its own Resend Request before processing the original Resend Request, potentially creating multiple overlapping recovery operations that require careful state management.

Complete Session Flow

Session Establishment

Connection Initiation:

INITIATOR ---> ACCEPTOR: Logon
8=FIX.4.4|9=89|35=A|49=CLIENT01|56=BROKER01|34=1|52=20251012-09:00:00.000|98=0|108=30|141=Y|553=user123|554=pass456|10=178|

ACCEPTOR ---> INITIATOR: Logon Response  
8=FIX.4.4|9=78|35=A|49=BROKER01|56=CLIENT01|34=1|52=20251012-09:00:00.123|98=0|108=30|141=Y|10=156|

(View in FixParser)

Active Trading Period

Business messages flow bidirectionally with incrementing sequence numbers. Each party independently maintains outbound sequence counters and expected inbound sequence counters.

Idle Period Heartbeat

During periods without business messages:

CLIENT01 ---> BROKER01: Heartbeat (sequence 47)
8=FIX.4.4|9=58|35=0|49=CLIENT01|56=BROKER01|34=47|52=20251012-09:15:30.000|10=123|

Gap Detection and Recovery

Gap Detection Scenario:

CLIENT01 ---> BROKER01: Resend Request (detected gap: expected 23, received 25)
8=FIX.4.4|9=57|35=2|49=CLIENT01|56=BROKER01|34=48|52=20251012-09:16:00.000|7=23|16=24|10=145|

BROKER01 ---> CLIENT01: Resend Message 23 (with PossDupFlag and OrigSendingTime)
8=FIX.4.4|9=XXX|35=D|49=BROKER01|56=CLIENT01|34=23|52=20251012-09:16:05.000|43=Y|122=20251012-09:10:15.000|...|10=XXX|

BROKER01 ---> CLIENT01: Resend Message 24
8=FIX.4.4|9=XXX|35=8|49=BROKER01|56=CLIENT01|34=24|52=20251012-09:16:05.100|43=Y|122=20251012-09:10:45.000|...|10=XXX|

(View in FixParser)

Session Termination

Bidirectional Logout:

CLIENT01 ---> BROKER01: Logout
8=FIX.4.4|9=73|35=5|49=CLIENT01|56=BROKER01|34=49|52=20251012-17:00:00.000|58=End of trading day|10=145|

BROKER01 ---> CLIENT01: Logout Response
8=FIX.4.4|9=65|35=5|49=BROKER01|56=CLIENT01|34=156|52=20251012-17:00:00.123|58=Goodbye|10=123|

(View in FixParser)

Error Handling and Edge Cases

Sequence Number Violations

Gap Detected (Too High):
The receiver sends a Resend Request for the missing range and queues any newer messages received while waiting for gap fill. Messages are released to the application layer only after gaps are resolved.

Sequence Too Low (Duplicate or Replay):
When a message arrives with a sequence number lower than expected without PossDupFlag=Y, a serious protocol violation has occurred. The recipient must immediately terminate the session via Logout with an explanatory Text field. This scenario indicates potential state desynchronization that could result in duplicate order processing.

Test Request Mechanism

When heartbeat violations occur:

8=FIX.4.4|9=78|35=1|49=CLIENT01|56=BROKER01|34=50|52=20251012-15:45:30.000|112=HEARTBEAT_CHECK_001|10=189|

The Test Request (MsgType=1) probes for counterparty responsiveness. The counterparty must respond with a Heartbeat containing the echoed TestReqID. Failure to respond within a reasonable timeout (typically equal to the heartbeat interval) indicates connection failure and triggers logout or forced disconnection.

Memory-Efficient State Management

High-frequency trading environments require efficient sequence number state management. Implementations should avoid storing entire message histories in memory. Instead, maintain:

Current outbound sequence number (32-bit or 64-bit integer)
Expected inbound sequence number (32-bit or 64-bit integer)
Bounded queue of recently sent messages for potential resend (circular buffer with configurable depth)
Pending gap ranges awaiting resolution

This approach limits memory consumption while supporting gap recovery without unbounded storage requirements.

Out-of-Order TCP Segment Handling

During periods of high network load, TCP may deliver segments out of order at the application layer despite TCP's ordering guarantees. FIX implementations must buffer messages received with higher-than-expected sequence numbers and trigger Resend Requests only after a timeout period (typically 100-500 milliseconds) to distinguish genuine gaps from transient reordering.

Production Implementation Considerations

Transport Layer Security

Most production exchanges mandate TLS/SSL encryption for FIX connections. This requirement exists independently of the EncryptMethod field (tag 98), which controls FIX application-layer encryption. Certificate validation, cipher suite selection, and TLS version negotiation occur at the transport layer before any FIX messages are exchanged.

Session State Persistence

Implementations supporting sequence number continuation (ResetSeqNumFlag=N) must persist session state across application restarts:

Last sent sequence number
Last received sequence number
SenderCompID and TargetCompID bindings
Session start time

Database or file-based persistence ensures accurate state recovery after crashes or planned maintenance windows.

High-Frequency Performance Optimization

Sub-millisecond latency requirements demand:

Zero-copy message parsing when possible
Pre-allocated message buffers to avoid allocation overhead
Lock-free or low-contention data structures for sequence state
Direct TCP socket access bypassing kernel networking stacks where permitted
CPU affinity pinning for FIX processing threads

Production systems handling billions of daily messages require these optimizations to meet performance targets while maintaining protocol correctness.

Monitoring and Observability

Production FIX engines should emit metrics for:

Message throughput (messages per second, inbound and outbound)
Gap recovery frequency and duration
Heartbeat latency (time between expected and received heartbeats)
Sequence number lag (difference between expected and actual sequence numbers)
Authentication failure rates
Average message processing latency

These metrics enable operators to detect degradation before it impacts trading operations.

Conclusion

The FIX protocol's session management layer provides battle-tested infrastructure for reliable, ordered message delivery in high-stakes financial environments. The four core messages.Logon, Logout, Heartbeat, and Resend Request establish a framework that processes billions of messages daily with sub-millisecond latency requirements across thousands of global institutions.

The session layer's separation from application-layer business logic exemplifies sound architectural design. This clean separation allows trading systems to focus on order management, execution, and risk control while delegating transport reliability to the session layer.

Production FIX implementations demand attention to subtle details: checksum calculation algorithms, sequence number state management, graceful logout procedures with timeout handling, and nested gap recovery scenarios. These technical nuances distinguish robust production systems from naive implementations that fail under adverse network conditions or high message volumes.

What Most Java Developers Miss About HashMap - And Why It Might Be Slowing You Down

Mark Andreev — Fri, 10 Oct 2025 07:59:00 +0000

The Hidden Pitfall of HashMap’s Initial Capacity

If you’ve been working with Java for a while, chances are you’ve used HashMap without giving much thought to its inner workings. After all, it’s a staple collection class-one of those things you just expect to work.

But here’s the catch: misusing the initial capacity setting can lead to unexpected performance issues, even in welloptimized codebases. And the worst part? Most developers don’t even realize they’re making the mistake.

So, what’s going on here? And how did Java 19 finally fix the problem?

Let’s break it down.

Understanding HashMap's Parameters

At its core, a HashMap has two parameters that dictate how efficiently it stores data: initial capacity and load factor.

Capacity is simply the number of buckets available to store entries. Load factor controls when the hash table should grow.
Whenever the number of entries exceeds the product of the load factor and current capacity, Java rehashes the table, increasing the number of buckets - usually doubling them. This automatic resizing is great in theory but can introduce unnecessary overhead if developers don’t set the initial capacity correctly.

Why Default Initialization is Misleading

Here’s where the problem comes in: most developers assume that setting the initial capacity means they’re defining the number of key-value pairs their HashMap will hold. But that’s not the case.
Instead, the constructor defines the bucket count, which doesn’t directly map to the expected number of entries. Because of this, developers have spent years manually adjusting their calculations using various formulas - each with slightly different results.

Some common ways Java developers have estimated the right capacity include:

(int) (numMappings / 0.75f) + 1
(int) ((float) numMappings / 0.75f + 1.0f)
(numMappings * 4 + 2) / 3
(int) ((numMappings * 4L + 2L) / 3L)
(int) Math.ceil(numMappings / 0.75f)

This inconsistency has popped up even in official Java code take java.lang.module.Resolver::makeGraph, which also contains a flawed assumption about capacity (source).

How Java 19 Finally Fixed It

After years of developers reinventing the wheel, Java 19 introduced HashMap::newHashMap(int numMappings), which finally offers a standardized way to create a properly sized HashMap.
Here’s how it works:

public static <K, V> HashMap<K, V> newHashMap(int numMappings) {
    if (numMappings < 0) {
        throw new IllegalArgumentException("Negative number of mappings: " + numMappings);
    }
    return new HashMap<>(calculateHashMapCapacity(numMappings));
}

static final float DEFAULT_LOAD_FACTOR = 0.75f;

static int calculateHashMapCapacity(int numMappings) {
    return (int) Math.ceil(numMappings / (double) DEFAULT_LOAD_FACTOR);
}

Instead of manually tweaking capacity values, developers can now call this method and let Java handle the calculation. This ensures that the map is correctly sized and avoids unnecessary resizing operations.

The update was part of JDK-8186958, and the full implementation can be found in this commit.

Best Practices Before Java 19

If you’re working with a Java version before 19, you have a couple of options:

Implement your own helper method similar to newHashMap to ensure correctly sized maps.
Prevent incorrect constructor usage across your team using code quality tools.

Using ArchUnit for Code Quality

One way to enforce best practices is with ArchUnit, which allows you to restrict incorrect constructor calls across your codebase.
The following rule ensures that developers don’t use new HashMap<>(capacity), forcing them to call HashMap.newHashMap(numMappings) instead:

import com.tngtech.archunit.base.DescribedPredicate;
import com.tngtech.archunit.core.domain.JavaConstructorCall;
import com.tngtech.archunit.junit.AnalyzeClasses;
import com.tngtech.archunit.junit.ArchTest;
import com.tngtech.archunit.lang.ArchRule;

import java.util.HashMap;

import static com.tngtech.archunit.lang.syntax.ArchRuleDefinition.noClasses;

@AnalyzeClasses(packages = "org.example")
public class HashMapCapacityConstructorRulesTest {
    @ArchTest
    static final ArchRule no_class_should_call_hashmap_capacity_constructor =
        noClasses().should().callConstructorWhere(new DescribedPredicate<>("Should not call HashMap<>(capacity). Use HashMap.newHashMap instead") {
            @Override
            public boolean test(JavaConstructorCall constructorCall) {
                return constructorCall.getTarget().getOwner().isEquivalentTo(HashMap.class)
                    && constructorCall.getTarget().getRawParameterTypes().size() == 1
                    && constructorCall.getTarget().getRawParameterTypes().getFirst().isEquivalentTo(int.class);
            }
        });
}

This helps teams prevent misleading API usage, ensuring that their projects are future-proof and scalable.

Conclusion: Why Clear API Design Matters

Sometimes, even fundamental Java classes contain misleading API designs. While using the wrong HashMap constructor may not always cause obvious performance bottlenecks, it’s still better to adopt modern best practices to avoid hidden inefficiencies.

With Java 19 introducing newHashMap, developers can finally move away from unclear capacity calculations. And for those still using older versions, enforcing proper usage with tools like ArchUnit is an easy win for code quality.

At the end of the day, clear API design isn’t just about performance - it’s about making code easier to understand, maintain, and evolve.

Vaadin Flow for AdminUI

Mark Andreev — Tue, 25 Feb 2025 22:54:19 +0000

Vaadin Flow is a server-driven UI framework that can serve as an alternative to React or Angular, where the frontend exists as a separate codebase written in a different programming language, with a clear boundary for data management (AJAX). For admin UIs, where backend developers manage the interface without deep knowledge of frontend technologies, this approach simplifies the architecture and reduces the overall maintenance effort. I would like to share my experience of adopting this framework.

Context

We are specifically dealing with admin UIs, typically have fewer concurrent users but each may have long-running sessions. An example of admin ui is social media moderator room where target users of ui are employees of a company that usually have long sessions. Another major attribute of admin ui is a complex UI with rich business logic, often making it unclear how to split code between frontend and backend.

All of this allows us to:

Manage the user's UI state on the backend since we can allocate resources for each user.
Put functionality ahead of perfect UX, as interface users can be trained.
Ignore initial loading time, as it is not critical for an admin UI.
Keep UI-related code alongside the backend, since they are tightly coupled, and no other consumers rely on the same API.

We also prefer open-source solutions with free-to-use licenses, as they provide the flexibility to fix critical bugs when needed.

Solution

All of this leads us to consider backend-driven UI frameworks like Vaadin Flow, a Java framework for server-rendered UIs. The majority of UI configuration—both layout and behavior—will be written in Java, just like the backend code.

For UI pages Vaadin flow provides Routes that can interact directly with spring services without a controllers layer. All state from Vaadin flow route transfers to the user browser using Vaadin internals without extra boilerplates like ajax requests and react subjects.

For special cases when integration with 3rd party libraries are required there are bridge interfaces with react (details).
UI testing in the free version might be done using Microsoft Playwright or Selenium.

Architecture

Enterprise applications often share similar requirements, which are implemented using common building blocks. Since Vaadin is designed for admin UIs and has undergone multiple iterations, you can find well-documented features in its official documentation. While some advanced components are available only in the paid version, with experience and guidance (even from ChatGPT), you can build them yourself.

Build (maven / gradle)
Assemble to container (docker)
Integration with beans management framework (Spring, Quarkus, JBoss CDI)
Secure using SSO (Authz)
Shared layout
Display data grids (tables)
Display forms with data

Build

Vaadin supports maven and gradle but documentation is well written only for maven. In some cases you may find bugs caused by gradle groovy to kotlin migration.

Containerisation

Vaadin offers a development mode with reduced build times and a production mode that compacts the JavaScript bundle and integrates user-defined components. It’s essential to test your production build to catch any potential build-related bugs early.

This topic covered by this guide (https://vaadin.com/docs/latest/flow/production/production-build )

Integration with beans management framework

Vaadin can be easily integrated with leading Java frameworks like Spring, Quarkus, and CDI. In the case of Spring, Vaadin routes can be treated as Spring components with a web session-like scope, meaning a bean is created per user session and retains state throughout user interactions. This scope is particularly intuitive for form management (covered in next sections).

Authz

While “SSO Kit” is a part of paid version you can rely on Spring authz that provides similar functionality. For example, you can delegate this task to Spring security with Keycloak.

spring.security.oauth2.client.registration.keycloak.client-id=${keycloak.client-id}
spring.security.oauth2.client.registration.keycloak.scope=openid,profile,email
spring.security.oauth2.client.registration.keycloak.authorization-grant-type=authorization_code
spring.security.oauth2.client.registration.keycloak.redirect-uri=${app.protocol}://${app.host} /login/oauth2/code/{registrationId}
spring.security.oauth2.client.provider.keycloak.issuer-uri=${keycloak.issuer-url}
spring.security.oauth2.client.provider.keycloak.jwk-set-uri=${keycloak.base-url}/protocol/openid-connect/certs

Layouts

As a starting point for hands-on experience, you can use Vaadin Start, which provides an online UI designer and generates related code. Take a closer look at the generated code to better understand AppLayout, which serves as a top-level placeholder for other views, and an example of a user-defined route (view) that extends VerticalLayout or HorizontalLayout.

Your UI is primarily built using two key layout components: HorizontalLayout and VerticalLayout, both of which can contain other UI components or nested layouts.

Data grids

Vaadin calls it com.vaadin.flow.component.grid.Grid (https://vaadin.com/docs/latest/components/grid ). It contains sorting, filtering and lazy loading.

Forms

The primary integration layer is com.vaadin.flow.data.binder.Binder, which synchronizes Java Beans with the UI state. It also provides an API for validation using withValidator.

// example
var binder = new Binder<>(BusinessPerson.class);

TextField nameField = new TextField();
binder.bind(nameField, BusinessPerson::getName, BusinessPerson::setName);

This topic covered by https://vaadin.com/docs/latest/flow/binding-data

Bonus: long running tasks

You might encounter a situation where each user interaction with a Java service, such as one involving long blocking database interactions, also blocks the user interface. While this may not be a problem for short interactions, long-running tasks, like bulk imports or exports, should be offloaded to threads that are not dedicated to rendering the UI. This approach is like what is done in other rendering frameworks, such as Swing or AWT.

More details https://vaadin.com/docs/latest/building-apps/application-layer/background-jobs , https://vaadin.com/docs/latest/flow/advanced/long-running-tasks

Alternatives

Since Vaadin is Java-focused, its major benefits are best realized within that ecosystem. If you're using .NET, Blazor might be a better fit, while in the Python world, a lightweight alternative could be Streamlit.

Author is Mark Andreev, SWE

What is RocksDB (and its role in streaming)?

Mark Andreev — Mon, 13 May 2024 08:25:51 +0000

RocksDB, a high-performance database, is a hidden gem in the tech industry, often overlooked by developers. I’m excited to delve into the intricacies and applications of this powerful tool. Who knows? It might just become the cornerstone of your next project.

Introduction

Originating from Facebook, RocksDB emerged as a potent storage engine designed for server workloads across diverse storage media, with an initial emphasis on high-speed storage, particularly Flash storage. It’s a C++ library that stores keys and values as byte streams of arbitrary sizes. It’s capable of both point lookups and range scans, and offers various ACID guarantees.

RocksDB strikes a harmonious balance between customizability and self-adaptability. It boasts highly adjustable configuration settings that can be fine-tuned to operate in a wide range of production environments, including SSDs, hard disks, ramfs, or remote storage. It supports a variety of compression algorithms and provides robust tools for production support and debugging. Simultaneously, efforts are made to limit the number of adjustable parameters, ensuring satisfactory out-of-box performance and implementing adaptive algorithms where suitable.

RocksDB incorporates substantial code from the open-source leveldb project and ideas from Apache HBase. The initial codebase was a fork from open-source leveldb. Furthermore, it builds upon code and concepts that were developed at Facebook prior to the inception of RocksDB.

Overview

The fundamental design principle of RocksDB is its performance optimization for fast storage and server workloads. It is engineered to facilitate efficient point lookups and range scans. Its configurability allows it to support high random-read workloads, high update workloads, or a blend of both. The architecture of RocksDB is designed to easily adjust trade-offs to cater to different workloads and hardware configurations.

RocksDB serves as a storage engine library, providing a key-value store interface where keys and values are represented as arbitrary byte streams. It arranges all data in a sorted sequence, and the typical operations include Get(key), NewIterator(), Put(key, val), Delete(key), and SingleDelete(key)

RocksDB does not natively support SQL. While it does not have a relational data model and does not support SQL queries, it can be used in conjunction with other systems or frameworks that provide SQL-like querying capabilities. For example, MyRocks combines RocksDB with MySQL. However, it’s important to note that RocksDB itself does not natively support SQL. It also does not have direct support for secondary indexes, but a user may build their own internally using Column Families or externally.

The main features are:

Designed for application servers wanting to store up to a few terabytes of data on local or remote storage systems.

Optimized for storing small to medium size key-values on fast storage -- flash devices or in-memory

It works well on processors with many cores

For read & write workload, rocksdb should be opened by a single process. RocksDB has the capability to support a multi-process read-only operation without any write operations to the database. This can be achieved by invoking the DB::OpenForReadOnly() method to open the database.

RocksDB was initially released as an open-source project under the BSD 3-clause license. However, in July 2017, the licensing of RocksDB was changed to a dual license model, incorporating both Apache 2.0 and GPLv2. This change was significant as it enabled RocksDB’s integration into projects under the Apache Software Foundation, which had previously blacklisted the BSD+Patents license clause. The dual-license information is available in the root directory of the RocksDB repository on GitHub. For the most recent updates, you may want to check the official RocksDB GitHub repository.

Applications

RocksDB is widely used in stream processing frameworks like Apache Flink, serving as a fast and efficient state store for maintaining the state of streaming applications. It’s also utilized in web applications for caching and session storage due to its efficient memory utilization and fast read/write operations. Additionally, RocksDB serves as a storage engine in database systems like TiKV that require high performance and durability, and is used in embedded systems such as IoT devices or edge computing applications due to its small memory footprint and light design.

What is Streaming

Streaming and batching are two different methods of processing data, each with its own use cases and advantages.

Streaming is a data processing method where each data item is processed individually and in real time as soon as it arrives. It’s like a water stream, where data flows continuously. This method is particularly useful when you need to process live data, such as real-time analytics, live monitoring systems, or real-time recommendations. The advantage of streaming is that it provides real-time or near-real-time insights. However, it can be more complex to implement and requires a robust infrastructure to handle continuous data flow.

On the other hand, batching is a method where data is collected over a period of time and processed all at once. It’s like filling a bucket with water and then processing all the water at once. This method is often used when the data doesn’t need to be processed in real time, such as daily reports, historical data analysis, or large ETL (Extract, Transform, Load) jobs. The advantage of batching is that it’s simpler to implement and can be more cost-effective, as it allows for resource optimization. However, it doesn’t provide real-time insights and can be slower, as you need to wait for the batch to be completed before you can process the data.

In summary, the choice between streaming and batching depends on your specific use case and requirements. If you need real-time insights, streaming is the way to go. If you’re dealing with large volumes of data and don’t need real-time processing, batching could be a more efficient choice. As a data engineer, it’s crucial to understand these differences to make the best decision for your data processing needs.

Streaming. Apache Flink

Applications that process streams often have a stateful nature, meaning they retain information from events they’ve processed to influence the processing of future events. In the context of Flink, this retained information, or state, is locally stored in a state backend that’s been configured. To safeguard against data loss during failures, the state backend periodically takes a snapshot of its contents and stores it in a durable storage that’s been pre-configured.
Among the three built-in state backends in Flink, the RocksDB state backend (also known as RocksDBStateBackend) is one.

You can find example of usage in org/apache/flink/contrib/streaming/state package (https://github.com/apache/flink/tree/9fe8d7bf870987bf43bad63078e2590a38e4faf6/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state).

Streaming. Kafka Streams & KSQL

Kafka Streams and KSQL use RocksDB as their default storage engine for stateful operations.In KSQL, RocksDB is used to store the materialized view locally on its disk. RocksDB is an embedded key/value store that runs in process in each KSQL server. You do not need to start, manage, or interact with it. This allows KSQL to execute stateful operations efficiently and reliably.

You can find example of usage in RocksDBStore class (https://github.com/a0x8o/kafka/blob/master/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java).

CockroachDB

CockroachDB is a distributed SQL database system developed by Cockroach Labs. It is designed to be resilient and consistent, much like its namesake, the cockroach. The system is known for its scalability, high availability, and versatility. It can handle increasing loads by adding more nodes, ensures data is always accessible, and can be run in various environments.

One of the key features of CockroachDB is its compatibility with PostgreSQL. It supports the PostgreSQL wire protocol and the majority of PostgreSQL syntax. This means that existing applications built on PostgreSQL can often be migrated to CockroachDB without changing application code. However, it’s important to note that CockroachDB does not support some of the PostgreSQL features or behaves differently from PostgreSQL because not all features can be easily implemented in a distributed system.

CockroachDB uses RocksDB as its underlying storage engine. The architecture of CockroachDB is divided into two layers: the SQL layer and the storage layer. The SQL layer sits on top of a transactional and strongly-consistent distributed key-value store.

In this key-value store, the key ranges are divided and stored in RocksDB. This allows CockroachDB to leverage the efficient storage mechanism of RocksDB while providing the benefits of a SQL interface. The data stored in RocksDB is replicated across the cluster, enhancing the resilience of the system. This design choice enables CockroachDB to handle large-scale, distributed storage while maintaining high performance and strong consistency.

You can find details about RocksDB usage in CockroachDB at their blog https://www.cockroachlabs.com/blog/cockroachdb-on-rocksd/

TiKV

TiKV, short for “Ti Key-Value”, is an open-source, distributed, and transactional key-value database. It’s designed to handle vast amounts of data while ensuring strong consistency and reliability. Unlike traditional NoSQL systems, TiKV provides both classical key-value APIs and transactional APIs with ACID compliance.

Built in Rust and powered by the Raft consensus algorithm, TiKV was originally created by PingCAP to complement TiDB, a distributed HTAP (Hybrid Transactional and Analytical Processing) database compatible with the MySQL protocol. The design of TiKV is inspired by some great distributed systems from Google, such as BigTable, Spanner, and Percolator, and some of the latest achievements in academia in recent years, such as the Raft consensus algorithm.

TiKV uses RocksDB as its primary storage layer. This design choice allows TiKV to handle large-scale, distributed storage while maintaining high performance and strong consistency.

In TiKV’s architecture, data is stored as key-value pairs. The keys are divided into ranges, and each range of keys is stored in RocksDB. This allows TiKV to leverage the efficient storage mechanism of RocksDB.

TiKV also uses the Raft consensus algorithm for data replication. For every write request, TiKV first writes the request to the Raft log. After the log is committed, TiKV applies the Raft log and writes the data to RocksDB. This ensures data consistency across multiple replicas in the cluster.

Moreover, TiKV uses a feature of RocksDB called Prefix Bloom Filter (PBF). PBF can filter out data which is impossible to contain keys with the same prefix as the row key provided. This is particularly useful in TiKV’s multi-version concurrency control (MVCC) model, where multiple versions of the same row share the same prefix. This feature enhances the efficiency of read operations.

In summary, RocksDB plays a crucial role in TiKV’s ability to provide a reliable, high-performance distributed storage solution.

Apache Kvrocks

Apache Kvrocks, a distributed key-value NoSQL database, utilizes RocksDB as its storage engine and is designed to be compatible with the Redis protocol.

Kvrocks has the following key features:

Redis Compatible: Users can access Apache Kvrocks via any Redis client.

Namespace: Similar to Redis SELECT but equipped with token per namespace.

Replication: Async replication using binlog like MySQL.

High Availability: Support Redis sentinel to failover when master or slave was failed.

Cluster: Centralized management but accessible via any Redis cluster client.

You can find details in their official documentation https://kvrocks.apache.org/community/data-structure-on-rocksdb/

Use in application

RocksDB is an embedded database, which means it is designed to be integrated directly into your application. Unlike standalone database systems, an embedded database like RocksDB operates as an integral part of your application, allowing for efficient data management and storage within the application itself. This can lead to improved performance and easier data handling, as the database operations are closely tied to the application’s functionality. However, it also means that the application is responsible for the direct management of the database.

C++

In C++ you can operate with database using:

// Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
//  This source code is licensed under both the GPLv2 (found in the
//  COPYING file in the root directory) and Apache 2.0 License
//  (found in the LICENSE.Apache file in the root directory).

#include <cstdio>
#include <string>

#include "rocksdb/db.h"
#include "rocksdb/options.h"
#include "rocksdb/slice.h"

using ROCKSDB_NAMESPACE::DB;
using ROCKSDB_NAMESPACE::Options;
using ROCKSDB_NAMESPACE::PinnableSlice;
using ROCKSDB_NAMESPACE::ReadOptions;
using ROCKSDB_NAMESPACE::Status;
using ROCKSDB_NAMESPACE::WriteBatch;
using ROCKSDB_NAMESPACE::WriteOptions;

#if defined(OS_WIN)
std::string kDBPath = "C:\\Windows\\TEMP\\rocksdb_simple_example";
#else
std::string kDBPath = "/tmp/rocksdb_simple_example";
#endif

int main() {
  DB* db;
  Options options;
  // Optimize RocksDB. This is the easiest way to get RocksDB to perform well
  options.IncreaseParallelism();
  options.OptimizeLevelStyleCompaction();
  // create the DB if it's not already present
  options.create_if_missing = true;

  // open DB
  Status s = DB::Open(options, kDBPath, &db);
  assert(s.ok());

  // Put key-value
  s = db->Put(WriteOptions(), "key1", "value");
  assert(s.ok());
  std::string value;
  // get value
  s = db->Get(ReadOptions(), "key1", &value);
  assert(s.ok());
  assert(value == "value");

  // atomically apply a set of updates
  {
    WriteBatch batch;
    batch.Delete("key1");
    batch.Put("key2", value);
    s = db->Write(WriteOptions(), &batch);
  }

  s = db->Get(ReadOptions(), "key1", &value);
  assert(s.IsNotFound());

  db->Get(ReadOptions(), "key2", &value);
  assert(value == "value");

  {
    PinnableSlice pinnable_val;
    db->Get(ReadOptions(), db->DefaultColumnFamily(), "key2", &pinnable_val);
    assert(pinnable_val == "value");
  }

  {
    std::string string_val;
    // If it cannot pin the value, it copies the value to its internal buffer.
    // The intenral buffer could be set during construction.
    PinnableSlice pinnable_val(&string_val);
    db->Get(ReadOptions(), db->DefaultColumnFamily(), "key2", &pinnable_val);
    assert(pinnable_val == "value");
    // If the value is not pinned, the internal buffer must have the value.
    assert(pinnable_val.IsPinned() || string_val == "value");
  }

  PinnableSlice pinnable_val;
  s = db->Get(ReadOptions(), db->DefaultColumnFamily(), "key1", &pinnable_val);
  assert(s.IsNotFound());
  // Reset PinnableSlice after each use and before each reuse
  pinnable_val.Reset();
  db->Get(ReadOptions(), db->DefaultColumnFamily(), "key2", &pinnable_val);
  assert(pinnable_val == "value");
  pinnable_val.Reset();
  // The Slice pointed by pinnable_val is not valid after this point

  delete db;

  return 0;
}

You can find details in official wiki in github https://github.com/facebook/rocksdb/wiki/Basic-Operations

Java

RocksJava is an initiative aimed at developing a Java driver for RocksDB that is both high-performing and user-friendly.

<dependency>
  <groupId>org.rocksdb</groupId>
  <artifactId>rocksdbjni</artifactId>
  <version>6.6.4</version>
</dependency>

You can open a database using

import org.rocksdb.RocksDB;
import org.rocksdb.RocksDBException;
import org.rocksdb.Options;
...
  // a static method that loads the RocksDB C++ library.
  RocksDB.loadLibrary();

  // the Options class contains a set of configurable DB options
  // that determines the behaviour of the database.
  try (final Options options = new Options().setCreateIfMissing(true)) {

    // a factory method that returns a RocksDB instance
    try (final RocksDB db = RocksDB.open(options, "path/to/db")) {

        // do something
    }
  } catch (RocksDBException e) {
    // do some error handling
    ...
  }
...

And do some read and write operations:

byte[] key1;
byte[] key2;
// some initialization for key1 and key2

try {
  final byte[] value = db.get(key1);
  if (value != null) {  // value == null if key1 does not exist in db.
    db.put(key2, value);
  }
  db.delete(key1);
} catch (RocksDBException e) {
  // error handling
}

You can find details in official documentation https://github.com/facebook/rocksdb/wiki/RocksJava-Basics

Conclusion

In conclusion, RocksDB is an excellent solution for managing persistent state on disk due to its high performance and efficient key-value storage. It’s particularly well-suited for applications that require fast, low-latency access to disk-based data. However, if your needs extend to larger-scale distributed state management or constraint management, more abstract databases such as MySQL might be a better fit. MySQL, being a relational database, offers features like transactional support, advanced querying capabilities, and robustness in handling complex data relationships, making it more suitable for these more complex scenarios. Therefore, the choice between RocksDB and MySQL (or any other database) should be guided by the specific requirements of your application.

Author is Mark Andreev, SWE @ Conundrum.AI

How to create Unit tests for code design?

Mark Andreev — Mon, 15 Apr 2024 08:56:20 +0000

As a seasoned software engineer, I understand the importance of well-designed code and the role of unit tests in maintaining its integrity. Code design is the foundation of any software system. It's the blueprint that guides the construction of the system, ensuring that the final product is robust, scalable, and maintainable. A well-designed codebase is easy to understand, modify, and extend, making it resilient to the ever-changing demands of the business environment.

Consistent code design is crucial for a team's productivity and the long-term maintainability of a project. However, achieving it is not a trivial task. It requires clear communication, shared understanding, and discipline from all team members. The difficulty lies in the fact that each developer has their unique coding style and problem-solving approach. Over time, without a concerted effort to maintain consistency, the codebase can become a patchwork of different styles and patterns, making it hard to understand and maintain.

I would like to share an approach based on ArchUnit - unit tests for your code.

ArchUnit is a powerful library developed by TNG that brings architectural control right into your codebase. The core idea of the project is to validate architectural rules in your Java codebase, ensuring that the design principles and standards you’ve set are being adhered to.

The library provides a domain-specific language (DSL) that allows you to express architectural rules in a clear, concise manner. This makes it easier for developers to understand and enforce these rules, leading to a more consistent and maintainable codebase.

@AnalyzeClasses(packages = "com.tngtech.archunit.example.layers")
public class MethodsTest {
    @ArchTest
    static ArchRule codeUnitsInDAOLayerShouldNotBeSecured =
            noCodeUnits()
                    .that().areDeclaredInClassesThat().resideInAPackage("..persistence..")
                    .should().beAnnotatedWith(Secured.class);
}

One of the main benefits of using ArchUnit is its ability to catch architectural violations early. Instead of waiting for code reviews or architectural audits, ArchUnit can identify issues as soon as the code is written. This leads to quicker feedback and less technical debt.

Furthermore, ArchUnit is flexible and extensible, allowing you to define custom rules that fit your project’s unique needs. It integrates seamlessly with popular testing frameworks like JUnit, making it a natural fit for any Java project.

For example you can disable JUnit Assertions when you prefer AssertJ library:

@Test
  void isTestClassesDontUseJunitAssertions() {
    ArchRuleDefinition.noClasses()
        .should()
        .dependOnClassesThat()
        .haveNameMatching("org.junit.jupiter.api.Assertions")
        .check(importedTestClasses);
  }

Or for Spring you can add restrictions for Controller names:

@Test
  void controllerNameRule() {
    classes()
        .that()
        .areAnnotatedWith(RestController.class)
        .should()
        .haveSimpleNameContaining("Controller")
        .check(importedClasses);
  }

Or even check Validation usage in RequestBody:

@Test
  void restControllerValidationRule() {
    classes()
        .that()
        .areAnnotatedWith(RestController.class)
        .should()
        .beAnnotatedWith(Validated.class)
        .check(importedClasses);
  }

@Test
  void restControllerValidationRequestBodyRule() {
    classes()
        .that()
        .areAnnotatedWith(RestController.class)
        .should(
            new ArchCondition<>("Any @RequestBody must be @Valid") {
              @Override
              public void check(JavaClass javaClass, ConditionEvents conditionEvents) {
                for (JavaMethod method : javaClass.getMethods()) {
                  if (method.isConstructor()) {
                    continue;
                  }

                  for (Parameter parameter : method.reflect().getParameters()) {
                    if (parameter.isAnnotationPresent(RequestBody.class)
                        && !parameter.isAnnotationPresent(Valid.class)) {
                      conditionEvents.add(
                          new SimpleConditionEvent(
                              javaClass,
                              false,
                              javaClass.getName()
                                  + " contains method "
                                  + method
                                  + " with @RequestBody and without Valid"));
                    }
                  }
                }
              }
            })
        .check(importedClasses);
  }

Absolutely, one of the powerful features of ArchUnit is its predefined rules. These rules provide a solid foundation for enforcing architectural standards in your Java codebase. They cover a wide range of common architectural scenarios, making it easier for you to get started with architectural control.

These predefined rules can be used as-is, or they can serve as a starting point for creating your own custom rules. This flexibility allows you to tailor ArchUnit to the specific needs of your project, ensuring that your architectural standards are upheld in a way that makes sense for your team and your codebase.

In essence, ArchUnit’s predefined rules are a valuable tool for maintaining architectural integrity, reducing technical debt, and enhancing the overall quality of your code. They are a testament to the library’s power and versatility, making it an essential tool for any Java developer serious about architecture.

You can found general predefined rules in com.tngtech.archunit.library.GeneralCodingRules like:

NO_CLASSES_SHOULD_ACCESS_STANDARD_STREAMS
NO_CLASSES_SHOULD_THROW_GENERIC_EXCEPTIONS
NO_CLASSES_SHOULD_USE_JODATIME
NO_CLASSES_SHOULD_USE_JAVA_UTIL_LOGGING
NO_CLASSES_SHOULD_USE_FIELD_INJECTION

And dependency rules available in com.tngtech.archunit.library.DependencyRules:

NO_CLASSES_SHOULD_DEPEND_UPPER_PACKAGES

More examples you can find at official user guide https://www.archunit.org/userguide/html/000_Index.html or at github repository https://github.com/TNG/ArchUnit

In summary, ArchUnit is a valuable tool for maintaining the integrity of your codebase’s architecture, promoting consistency, reducing technical debt, and enhancing overall code quality. It’s a must-have for any team serious about architecture.

Author is Mark Andreev, SWE @ Conundrum.ai

Enhance Your Project Quality with These Top Python Libraries

Mark Andreev — Mon, 18 Mar 2024 14:15:00 +0000

In the ever-evolving landscape of software development, Python has emerged as a language of choice for programmers worldwide. Its simplicity, versatility, and robust library support have made it a favourite among both beginners and seasoned developers. This article, titled “Enhance Your Project Quality with These Top Python Libraries”, aims to guide you through some of the most powerful and efficient Python libraries that can significantly enhance the quality of your projects.

Python’s extensive collection of libraries is one of its most attractive features. These libraries, each with its unique capabilities, can simplify complex tasks, reduce coding time, and increase the efficiency and performance of your projects. Whether you’re working on data analysis, machine learning, web development, or any other domain, Python has a library that can make your job easier. Please note that while this article aims to be as comprehensive and accurate as possible, the world of Python libraries is vast and constantly evolving. Therefore, I encourage readers to continue exploring and experimenting beyond what is covered here.

In the realm of software development, the quality of code is paramount. It is the bedrock upon which reliable, efficient, and maintainable software is built. The main components that contribute to code quality include code style, linters, code formatters, type checking, unit tests, and documentation.

Code style

Code style is the set of conventions and guidelines that developers adhere to when writing code. It ensures consistency, readability, and understandability across the codebase, making it easier for others to read and maintain. Code formatters automatically format code according to a specified style guide. They save developers time and effort, and ensure that the codebase has a consistent look and feel.

There are several Python libraries that can help improve code style:

PEP8 (pycodestyle): Named after Python’s PEP 8 style guide, this tool checks your Python code against some of the style conventions in PEP 8.
Black: Known as “The Uncompromising Code Formatter”, Black automatically formats your Python code to conform to the PEP 8 style guide. It takes away the hassle of having to manually adjust your code style.
isort: This library sorts your imports alphabetically, and automatically separates them into sections and by type. It provides a cleaner and more organised way to manage project imports.
YAPF (Yet Another Python Formatter): YAPF takes a different approach in that it’s based off of ‘clang-format’, a popular formatter for C++ code. YAPF reformats Python code so that it conforms to the style guide and looks good.
AutoPEP8: This tool automatically formats Python code to conform to the PEP 8 style guide. It uses pycodestyle, a library that encapsulates the functionality of the original pep8 tool.

These libraries can be integrated into your development environment or used as part of your continuous integration pipeline to ensure that all code adheres to your project’s style guidelines.

Linters

Linters are tools that analyse code to detect potential errors, bugs, stylistic errors, and suspicious constructs. They enforce a level of quality and consistency, and help developers avoid common pitfalls and mistakes.

Linting is an essential part of writing clean, error-free Python code. Here are some Python linter libraries that are widely used:

Pylint. This is one of the most popular and comprehensive linter tools for Python. Pylint checks for errors, tries to enforce coding standards, and looks for bad smells in your code. It also provides useful features such as checking line-code’s length, checking if variable names are well-formed according to your coding standard, and more.
Flake8. This library is a wrapper around pycodestyle (PEP8), pyflakes, and Ned Batchelder’s McCabe script. It is a great toolkit for checking your code base against coding style (PEP8), programming errors (like SyntaxError, NameError, etc) and to check cyclomatic complexity.
Pyflakes is a simple program that checks Python source files for errors. It is similar to PyLint but focuses only on errors, not style. This makes it faster and less intrusive than some other tools.
Bandit is a tool designed to find common security issues in Python code. It was developed by the OpenStack Security Project and is a great addition to any serious Python project.
Ruff is a Python linter that helps to identify and remove code smells. Over 700 built-in rules: Ruff includes native re-implementations of popular Flake8 plugins, like flake8-bugbear. And also built-in caching to avoid re-analyzing unchanged files.
Pylama is a code audit tool for Python that wraps tools like Pylint, pycodestyle, PyFlakes, McCabe, and others to provide a unified interface.
Prospector is a Python static analysis tool that combines multiple Python linters for a more comprehensive error check.

These linter libraries can be integrated into your development environment or used as part of your continuous integration pipeline to ensure that all code adheres to your project’s style guidelines and is free from common errors.

Type checking

Type checking is the process of verifying and enforcing the constraints of types. It can catch and prevent many types of bugs, making the code safer and more predictable.

Mypy is a static type checker for Python that combines the benefits of dynamic typing and static typing. Mypy checks that the types of variable values match their declared types, and it can catch a whole class of bugs before the code is even run.
Pyright is a fast type checker meant for large Python source bases. It can run in a “watch” mode and performs fast incremental updates when files are modified.
Pytype checks and infers types for your Python code - without requiring type annotations. Pytype can catch type errors in your Python code before you even run it.
Pyre is a performant type-checker developed by Facebook. Pyre can analyse codebases with millions of lines of code incrementally – providing instantaneous feedback to developers as they write code.
MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or add type annotations directly to your Python code based on the types collected at runtime.

Each of these libraries has its own strengths and weaknesses, and the choice between them often depends on the specific needs of the project. It’s also worth noting that these tools are most effective when used in conjunction with good testing practices.

Unit tests

Unit tests are small, isolated tests that verify the functionality of a specific section of code. They provide a safety net, allowing developers to refactor and add features with confidence.

PyTest is a mature full-featured Python testing tool that helps you write better programs. It provides a simple and easy way to write tests and has a powerful set of features. PyTest allows for compact test suites and introduces the concept that Python tests should be plain Python functions instead of forcing developers to include their tests inside large test classes.
Unittest is a built-in module of Python. It’s inspired by the xUnit framework architecture. This is a great tool to create and organise test cases in a systematic way. You can use unittest.mock with pytest when you need to create mock objects in your tests. The unittest.mock module is a powerful feature in Python’s standard library for creating mock objects in your tests. It allows you to replace parts of your system under test and make assertions about how they have been used. So, even though pytest is a different testing framework, it can work seamlessly with unittest.mock for creating and using mock objects.
DocTest is another module that’s part of the Python standard library. It searches for pieces of text that look like interactive Python sessions in docstrings, and then executes those sessions to verify that they work exactly as shown.
Tox is a powerful tool in Python development, primarily used for automating and standardised testing across multiple environments. It’s part of a larger vision of easing the packaging, testing, and release process of Python software. You should consider using tox in the following scenarios if your application needs to run across different Python versions, tox can help ensure your code works consistently across all these versions; If your project has complex dependencies, tox can manage and isolate these dependencies in different environments.
Hypothesis is a Python library for creating unit tests that are simpler to write and more powerful when run, finding edge cases in your code you wouldn’t have thought to look for. It is stable, powerful, and easy to add to any existing test suite. It allows you to write tests that assert that something should be true for every case, not just the ones you happen to think of. This approach is often called property-based testing, and was popularised by the Haskell library Quickcheck.

Documentation

Documentation is the written text that accompanies the code, explaining how it works and how to use it. Good documentation makes the codebase accessible and understandable to others, and serves as a valuable reference.

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation. It was originally created for the Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well.
MkDocs is a fast, simple and downright gorgeous static site generator that’s geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file.
Pydoc is a module in Python that generates documentation for modules, scripts, and classes. Pydoc provides a way to access Python’s built-in documentation strings and provides an interface for viewing the documentation in a user-friendly format.
Read the Docs simplifies software documentation by automating building, versioning, and hosting of your docs for you.
Doxygen is a documentation generator, a tool for writing software reference documentation. The documentation is written within code, and is thus relatively easy to keep up to date. Doxygen can cross reference documentation and code, so that the reader of a document can easily refer to the actual code.

Remember that code quality is a continuous process. Regularly review and refactor your code to maintain high standards. Happy coding!

Author is Mark Andreev, SWE @ Conundrum.ai