DEV Community: Bruno Oliveira

Basic RAG app with Spring AI, Docker and Ollama

Bruno Oliveira — Thu, 01 Aug 2024 11:40:04 +0000

Large Language models are becoming smaller and better over time, and, today, models like Llama3.1 which has competing benchmark scores with GPT-3.5 Turbo can be easily run locally from your own computer.

This opens up endless opportunities to build cool stuff on top of this cutting-edge innovation, and, if you bundle together a neat stack with Docker, Ollama and Spring AI, you have all you need to architect production-grade RAG systems locally.

Technology Stack

We will build a basic RAG app with:

Streamlit as a basic frontend;
Spring AI version 1.0.0-M1;
Springboot 3.3.0;
Java 21;
Docker+compose, to use PGVector as a RAG database in a container and Ollama to serve local models;
Claude.ai as a pair programming assistant to spar ideas with;

Note that I've added claude.ai as part of the technology stack, because I believe, that now, if you're not using genAI tools to help you program and sketch out ideas, you will quickly fall behind the curve.
My take about using genAI as coding assistants is very simple: you need to know what you're doing to extract value from them, but, if you have a good mental model (that you need to devise by doing and learning and reading by yourself for many years; there's no replacement for this but doing it alone, by yourself, seeking deep understanding) you can move much faster than otherwise.

The app

The application will function as a chatbot that allows you to select different LLMs and upload documents, so you can “chat” with them.

When the document(s) are uploaded, you can then “query” them via text messages that can be “matched” with contents of said documents.

How does it really work?

But how does this work, really?
The answer might be a lot less exciting than you think: it uses plain, decades old concepts from the fields of databases and information retrieval, there's nothing fancy here.

The basic idea is like taking an open-book exam and being allowed to take notes with you:

You have a question you need an answer to;
You have access to a giant knowledge base of information (your entire book);
Your notes represent a distilled version of the knowledge in the book (this is called an embedding in RAG slang);
You then perform a similarity search on your notes: map your question to your notes, and together with your own knowledge produce an answer;

The place where this analogy falls short is in something that our brain does implicitly, but, in RAG systems, we need to do very explicitly: the “question” and your “notes” need to be in a similar format so they can be compared with each other: you'd compare a “distilled” version of your question with your “distilled” knowledge, represented by your notes.

Then, using classic similarity algorithms, like cosine similarity, dot product between vectors, even plain old full-text search (and others I don't know about), you can find results that are close enough to what you need to retrieve.

The flow is complete when these “very close matches”, after being retrieved from your RAG database are sent to an LLM as “context”, so that it can use that context to provide you with better answers.

The reason this is so effective is that by providing an LLM with this additional information "retrieved" from your own personal or private documents, you are supplementing the model with extra information that it wouldn't otherwise have, which greatly improves its accuracy.

Diving into the code

Since LLMs emerged as the new kid on the block, most of the popular ecosystems have jumped head first into it, offering support for working with LLMs and RAG databases directly. We will use Springboot together with Spring AI and Postgres as our “Knowledge Base”, by installing the pgvector extension which gives vector searching capabilities to postgres.

There are many aspects and moving parts to make this work together, and, understanding the foundational concepts is important, so, let's go step by step.

First, there was a RAG database

We start with the database:



  db:
    image: ankane/pgvector:latest
    environment:
      - POSTGRES_DB=ragdb
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
    networks:
      - app-network

This uses a docker service as part of a larger docker-compose file, which we will build incrementally as we go along, that sets up a postgres database with support for the pgvector extension. It also uses a dedicated init.sql file to perform database initialization by creating the necessary DB tables:



CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS hstore;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

-- Create the documents table
CREATE TABLE IF NOT EXISTS documents (
    id UUID DEFAULT uuid_generate_v4() PRIMARY KEY,
    document_name TEXT UNIQUE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    metadata JSONB
);

-- Create the vector_store table with reference to documents
CREATE TABLE IF NOT EXISTS vector_store (
    id UUID DEFAULT uuid_generate_v4() PRIMARY KEY,
    document_id UUID NOT NULL,
    content TEXT,
    metadata JSONB,
    embedding vector(1024),
    chunk_index INTEGER,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (document_id) REFERENCES documents(id) ON DELETE CASCADE
);

-- Create indexes
CREATE INDEX idx_documents_document_name ON documents(document_name);
CREATE INDEX idx_vector_store_document_id ON vector_store(document_id);
CREATE INDEX idx_vector_store_embedding ON vector_store USING HNSW (embedding vector_cosine_ops);

A lot happening here, so, let's distill the most important concepts:



CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS hstore;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

These three lines, while not strictly needed here, because the docker image already has them pre-installed, are included for completeness and these are needed to enable the RAG support of a standard Postgres distribution.

Vector adds support for vector similarity search algorithms, hstore, a data type for storing sets of key/value pairs within a single PostgreSQL value, which can come in handy to map, for example, documents to their corresponding metadata values, or even chunks.
And lastly, uuid-ossp provides utility functions to generate UUIDs using several known standard algorithms.

Then, we have the table creation:



-- Create the documents table
CREATE TABLE IF NOT EXISTS documents (
    id UUID DEFAULT uuid_generate_v4() PRIMARY KEY,
    document_name TEXT UNIQUE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    metadata JSONB
);

-- Create the vector_store table with reference to documents
CREATE TABLE IF NOT EXISTS vector_store (
    id UUID DEFAULT uuid_generate_v4() PRIMARY KEY,
    document_id UUID NOT NULL,
    content TEXT,
    metadata JSONB,
    embedding vector(1024),
    chunk_index INTEGER,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (document_id) REFERENCES documents(id) ON DELETE CASCADE
);

This requires some additional explanation: we create a relationship between a document and its embeddings, while adding metadata to both entities: a document can have many embeddings, so, we will see a @ManyToOne relationship in the document id column of the vector_store table in Java, when we define the hibernate entities.

I've decided to create a separate documents table besides the vector_store one because it's useful to keep a separate table to manage the relationship between the document(s) and their many embeddings. It's also crucial because by enforcing a uniqueness constraint on the document_name in the documents table and referencing that via id in our vector_store table, that represents the embeddings of document chunks (or potentially full documents) we gain referential integrity: if we try to upload similar documents we won't generate repeated embeddings to bloat the DB unnecessarily. Obviously, for documents with the exact same name whose contents have been changed, this would cause problems, but, this is still just testing the waters and seeing how it all ties together.

Finally, indexes come into play:



-- Create indexes
CREATE INDEX idx_documents_document_name ON documents(document_name);
CREATE INDEX idx_vector_store_document_id ON vector_store(document_id);
CREATE INDEX idx_vector_store_embedding ON vector_store USING HNSW (embedding vector_cosine_ops);

Indexes on embeddings are a whole different beast, as we need to somehow find ways to efficiently index data inside potentially dense vectors, which comes with efficiency, size, and speed challenges that are not seen in regular types. This post is the best I've read about this so far.

Then, `PGVectorStore` enters the chat

We have looked at the DB structure for our documents and embeddings, now it's time to look into how Spring AI helps us leverage this.

The new Spring AI module of the Springboot framework offers support for Vector Databases, namely for configuring Postgres as a vector store.

On startup, the PgVectorStore will attempt to install the required database extensions and create the required vector_store table with an index if not existing.

By default, as indicated in the URL above, the created vector_store table will have a dimension of 1536.
This seems to be a standard dimension for embedding vectors, used for example by OpenAI for its small embedding models, so, it's, in a sense, a sensible default.

However, we might want to have either smaller or larger embedding vector dimensions (PGVector supports up to 2000 dimensions), which can mean we will be using specific embedding models that are smaller in size (for instance, Voyage-2 embedding model from Anthropic, which has 1024 dimensions).

Because of this, we will see, soon, that we need to pass some special environment variables to our Springboot app to have more fine-grained control over how the vector_store default PGVectorStore "interface" table is created, and which embedding model will be used in our application.

We can manually configure PGVectorStore by exposing some Beans in a configuration class:



@Bean(name = "ragDataSource")
    public DataSource ragDataSource() {
        return DataSourceBuilder.create()
                .url(datasourceUrl)
                .username(dataSourceUsername)
                .password(dataSourcePassword)
                .driverClassName("org.postgresql.Driver")
                .build();
    }

    @Bean(name = "ragDB")
    public JdbcTemplate jdbcTemplate() {
        return new JdbcTemplate(ragDataSource());
    }

    @Bean
    public VectorStore vectorStore(
            @Qualifier("ragDB") JdbcTemplate jdbcTemplate,
            @Qualifier("ollamaEmbeddingModel") EmbeddingModel embeddingModel) {
        return new PgVectorStore(jdbcTemplate, embeddingModel);
    }

We will now focus our attention on configuring our Springboot app via a docker-compose file to tie this together.

Docker-compose based Springboot configuration

Here is our Springboot app configuration in a docker-compose file:



  backend:
    build: .
    ports:
      - "8080:8080"
    environment:
      - SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/ragdb
      - SPRING_DATASOURCE_USERNAME=postgres
      - SPRING_DATASOURCE_PASSWORD=password
      - SPRING_AI_OLLAMA_BASE_URL=http://ollama-llm:11434/
      - SPRING_AI_OLLAMA_CHAT_OPTIONS_MODEL=tinydolphin
      - SPRING_AI_OLLAMA_CHAT_OPTIONS_ALTERNATIVE_MODEL=tinyllama
      - SPRING_AI_OLLAMA_CHAT_OPTIONS_ALTERNATIVE_SECOND_MODEL=llama3.1
      - SPRING_AI_OLLAMA_EMBEDDING_OPTIONS_MODEL=mxbai-embed-large
      - SPRING_AI_VECTORSTORE_PGVECTOR_REMOVE_EXISTING_VECTOR_STORE_TABLE=true
      - SPRING_AI_VECTORSTORE_PGVECTOR_INDEX_TYPE=HNSW
      - SPRING_AI_VECTORSTORE_PGVECTOR_DISTANCE_TYPE=COSINE_DISTANCE
      - SPRING_AI_VECTORSTORE_PGVECTOR_DIMENSIONS=1024
    depends_on:
      - prepare-models
      - db
    volumes:
      - ollama_data:/root/.ollama
    networks:
      - app-network

This is the docker-compose service representing our Java Springboot app that gets built fully locally. The picture is not 100% complete yet, as we need to look at how we configure Ollama via docker-compose, but, we will get there.

The most critical env. variables are the first 5 and the last 5. The other ones are extra, added to allow runtime configuration of different LLMs.

We define data source credentials to ensure that our application knows about and can connect to our Postgres database, which will serve as the RAG database.

Then we configure the URL for the API that is "hosting" the LLM. Usually, these large language models are served via APIs and client code will just connect to them to be able to use them. We need a host, where the inference API is running, and we need to specify the model we want to use to do the inference itself:



 - SPRING_AI_OLLAMA_BASE_URL=http://ollama-llm:11434/
 - SPRING_AI_OLLAMA_CHAT_OPTIONS_MODEL=tinydolphin

Since we are building a RAG application, we need an "embedding model" besides only an inference API. Remember the "notes" for the open-book exam? If we want to do RAG, we need a way to "create the notes". This is done by specifying an embedding "model".



- SPRING_AI_OLLAMA_EMBEDDING_OPTIONS_MODEL=mxbai-embed-large

I used an open-source embedding model from Mixedbread that is offered via the Ollama project, as an embedding model.

Usually, current LLM providers of services, such as OpenAI, Anthropic, Amazon Bedrock, etc, tend to offer an inference API, that uses a large language model and an embeddings API, which is an API that uses another model that just does embeddings. This is that concept materialized in the Ollama project.

Finally, we configure the PGVectorStore abstraction offered by Spring AI on top of a pgvector-enabled postgres installation to tweak its defaults in a way that everything matches:



      - SPRING_AI_VECTORSTORE_PGVECTOR_REMOVE_EXISTING_VECTOR_STORE_TABLE=true
      - SPRING_AI_VECTORSTORE_PGVECTOR_INDEX_TYPE=HNSW
      - SPRING_AI_VECTORSTORE_PGVECTOR_DISTANCE_TYPE=COSINE_DISTANCE
      - SPRING_AI_VECTORSTORE_PGVECTOR_DIMENSIONS=1024

The key aspects are that we define the remove-existing-vector-store-table to true, so that we don't let the autowiring of PGVectorStore create the vector_store table for us in the postgres installation. Remember, the default embedding dimensions in there are 1536, and we need compatibility in the dimensions of the embedding model we are using, which has 1024 dimensions, that we define in the end.
We also define the index type as well as the distance metric to use when performing similarity searches.

Ollama docker container to pull the models so they are locally available

Here is the setup I used to pull the models (both the large language models as well as the embeddings models so they get downloaded to my local machine:



  ollama-llm:
    image: ollama/ollama:latest
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    networks:
      - app-network

  prepare-models:
    image: ollama/ollama:latest
    depends_on:
      - ollama-llm
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=http://ollama-llm:11434
    networks:
      - app-network
    entrypoint: >
      sh -c "
        echo 'Waiting for Ollama server to start...' &&
        sleep 10 &&
        echo 'Pulling tinydolphin...' &&
        ollama pull tinydolphin &&
        echo 'Pulling tinyllama...' &&
        ollama pull tinyllama &&
      echo 'Pulling llama3.1...' &&
      ollama pull llama3.1 &&
      echo 'Pulling embedding model...' &&
      ollama pull mxbai-embed-large &&
        echo 'Model preparation complete.'"

I use a shared volume between these two services to make sure that there are two things happening: in a preparatory step, I pull the LLMs I want to use and the embedding model, and they get stored in the ollama_data:/root/.ollama volume.
Then, at a "runtime" step, where the actual Ollama embedding models are used as defined above in the Springboot app confuration, they will be available because the runtime Ollama container will look into the same volume where the models were downloaded and things will work as expected, just like if the models would be hosted somewhere as remote API endpoints for embedding and completion generation.

Java code to tie it all together

Since we have created special tables besides the standard ones offered via PGVectorStore, we want to leverage them fully, so, we create dedicated Hibernate entities so we can query them:



@Entity
@Table(name = "documents")
@Data
@Builder
@AllArgsConstructor
public class DocumentEntity {
    @Id
    @GeneratedValue(generator = "uuid2")
    @GenericGenerator(name = "uuid2", strategy = "uuid2")
    @Column(columnDefinition = "UUID")
    private UUID id;

    @Column(name = "document_name", unique = true, nullable = false)
    private String documentName;

    @Column(name = "created_at")
    private Instant createdAt;

    @Column(name = "updated_at")
    private Instant updatedAt;

    @JdbcTypeCode(SqlTypes.JSON)
    @Column(columnDefinition = "jsonb")
    private Map<String, Object> metadata;

    public DocumentEntity() {}

    @PrePersist
    protected void onCreate() {
        createdAt = Instant.now();
        updatedAt = Instant.now();
    }

    @PreUpdate
    protected void onUpdate() {
        updatedAt = Instant.now();
    }
}

and:



@Entity
@Table(name = "vector_store")
@Data
@Builder
@AllArgsConstructor
public class VectorStoreEntity {
    @Id
    @GeneratedValue(generator = "uuid2")
    @GenericGenerator(name = "uuid2", strategy = "uuid2")
    @Column(columnDefinition = "UUID")
    private UUID id;

    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "document_id", nullable = false)
    private DocumentEntity document;

    @Column(name = "content")
    private String content;

    @JdbcTypeCode(SqlTypes.JSON)
    @Column(columnDefinition = "jsonb")
    private Map<String, Object> metadata;

    @Column(name = "embedding", columnDefinition = "vector(1024)")
    private double[] embedding;

    @Column(name = "chunk_index")
    private Integer chunkIndex;

    @Column(name = "created_at")
    private Instant createdAt;

    @Column(name = "updated_at")
    private Instant updatedAt;

    public VectorStoreEntity() {}

    @PrePersist
    protected void onCreate() {
        createdAt = Instant.now();
        updatedAt = Instant.now();
    }

    @PreUpdate
    protected void onUpdate() {
        updatedAt = Instant.now();
    }
}

Then, we can create a dedicated embedding service that uses our table structure to store the embeddings:



@Slf4j
@Service
@AllArgsConstructor
@ConditionalOnProperty("spring.datasource.url")
public class EmbeddingService {

    private VectorStoreRepository vectorStoreRepository;
    private DocumentRepository documentRepository;

    @Qualifier("ollamaEmbeddingModel")
    private EmbeddingModel embeddingModel;

    public void generateAndStoreEmbedding(String input, String originalFilename) {
        var document = DocumentEntity.builder()
                .documentName(originalFilename)
                .metadata(Map.of("category", "PUBLIC"))
                .build();
        var savedDocument = documentRepository.save(document);

        vectorStoreRepository.saveAll(List.of(VectorStoreEntity.builder()
                .id(savedDocument.getId())
                .content(input)
                .embedding(
                        embeddingModel.embed(input).stream().mapToDouble(v -> v).toArray())
                .metadata(document.getMetadata())
                .build()));

        log.info("Stored embedding for input: " + input);
    }
}

And, with this in place, we can now create a dedicated RAG service with which we can perform document similarity search:



@Service
@AllArgsConstructor
@ConditionalOnProperty("spring.datasource.url")
public class RagService {

    private ChatModel chatModel;

    private VectorStore vectorStore;

    private final Map<String, DocumentProcessor> documentProcessors;

    @Getter
    private final Set<String> uploadedFileNames = new HashSet<>();

    private final VectorStoreRepository vectorStoreRepository;

    public List<VectorStoreEntity> findAll(String name) {
        return vectorStoreRepository.findVectorStoreEntitiesByDocument_DocumentName(name);
    }

    public void processFiles(List<MultipartFile> files) {
        for (MultipartFile file : files) {
            var extension = file.getOriginalFilename().split("\\.")[1];
            DocumentProcessor documentProcessor = documentProcessors.get(extension);
            documentProcessor.processDocument(file);
            uploadedFileNames.add(file.getOriginalFilename());
        }
    }

    public Flux<ChatResponse> query(String question) {
        var similarDocuments = vectorStore.similaritySearch(SearchRequest.query(question)
                .withTopK(5)
                .withFilterExpression(new Filter.Expression(
                        Filter.ExpressionType.EQ, new Filter.Key("category"), new Filter.Value("PUBLIC"))));
        System.out.println("Retrieved " + similarDocuments.size() + " similar docs.");
        String context = similarDocuments.stream().map(Document::getContent).collect(Collectors.joining("\n"));

        String prompt = "Context: " + context + "\n\nQuestion: " + question + "\n\nAnswer:";

        System.out.println("The prompt is " + prompt);
        return chatModel.stream(new Prompt(prompt));
    }
}

with this in place, we are now equiped with a solid foundation to extend our app with whatever we want.

Some food for tought:

Adding dedicated metadata at the document level and chunk level to enrich the LLM prompt even further;
Always remember to program against interfaces where possible: the DocumentProcessor above is an interface that selects a dedicated document processor at runtime to perform distinct pre-processing operations to distinct documents before generating embeddings;
We can extend this with a "post-processing" step for the embeddings, like adding metadata if desired, etc;

Finally, here's the full docker-compose file which includes the streamlit front-end too:



version: '3.8'

services:

  ollama-llm:

    image: ollama/ollama:latest

    volumes:

      - ollama_data:/root/.ollama

    ports:

      - "11434:11434"

    networks:

      - app-network

db:

    image: ankane/pgvector:latest

    environment:

      - POSTGRES_DB=ragdb

      - POSTGRES_USER=postgres

      - POSTGRES_PASSWORD=password

    volumes:

      - postgres_data:/var/lib/postgresql/data

      - ./init.sql:/docker-entrypoint-initdb.d/init.sql

    networks:

      - app-network

prepare-models:

    image: ollama/ollama:latest

    depends_on:

      - ollama-llm

    volumes:

      - ollama_data:/root/.ollama

    environment:

      - OLLAMA_HOST=http://ollama-llm:11434

    networks:

      - app-network

    entrypoint: >

      sh -c "

        echo 'Waiting for Ollama server to start...' &&

        sleep 10 &&

        echo 'Pulling tinydolphin...' &&

        ollama pull tinydolphin &&

        echo 'Pulling tinyllama...' &&

        ollama pull tinyllama &&

      echo 'Pulling llama3.1...' &&

      ollama pull llama3.1 &&

      echo 'Pulling embedding model...' &&

      ollama pull mxbai-embed-large &&

        echo 'Model preparation complete.'"

backend:

    build: .

    ports:

      - "8080:8080"

    environment:

      - SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/ragdb

      - SPRING_DATASOURCE_USERNAME=postgres

      - SPRING_DATASOURCE_PASSWORD=password

      - SPRING_AI_OLLAMA_BASE_URL=http://ollama-llm:11434/

      - SPRING_AI_OLLAMA_CHAT_OPTIONS_MODEL=tinydolphin

      - SPRING_AI_OLLAMA_CHAT_OPTIONS_ALTERNATIVE_MODEL=tinyllama

      - SPRING_AI_OLLAMA_CHAT_OPTIONS_ALTERNATIVE_SECOND_MODEL=llama3.1

      - SPRING_AI_OLLAMA_EMBEDDING_OPTIONS_MODEL=mxbai-embed-large

      - SPRING_AI_VECTORSTORE_PGVECTOR_REMOVE_EXISTING_VECTOR_STORE_TABLE=true

      - SPRING_AI_VECTORSTORE_PGVECTOR_INDEX_TYPE=HNSW

      - SPRING_AI_VECTORSTORE_PGVECTOR_DISTANCE_TYPE=COSINE_DISTANCE

      - SPRING_AI_VECTORSTORE_PGVECTOR_DIMENSIONS=1024

    depends_on:

      - prepare-models

      - db

    volumes:

      - ollama_data:/root/.ollama

    networks:

      - app-network

frontend:

    build:

      context: ./frontend

      dockerfile: Dockerfile

    ports:

      - "8501:8501"

    environment:

      - BACKEND_URL=http://backend:8080

      - BACKEND_AUTH_USERNAME=admin

      - BACKEND_AUTH_PASSWORD=cst=stom

    depends_on:

      - backend

    networks:

      - app-network

networks:

  app-network:

    driver: bridge

volumes:

  postgres_data:

  ollama_data:

Conclusion

With Spring AI we can leverage all the latest advancements in the LLM and AI research fields and, together with concepts from DB, information retrieval and data representation, we can build full-fledged local RAG apps that are modular and easy to extend!

Announcement: Book release!!

Bruno Oliveira — Tue, 26 Oct 2021 11:50:03 +0000

As the title says:

Inspired by my series of Flask posts, I decided to structure them more, add a lot of details regarding the deployment process, and, as a result, I have just released my first book:

https://brunooliv.gumroad.com/l/YAjWX

Parsing and reading Excel binary data using Pandas

Bruno Oliveira — Sat, 23 Oct 2021 12:55:57 +0000

Introduction

Sometimes, it can be necessary to parse data in binary format from an external source in order to work with it.
Usually, when encoding the data from its original source into a binary format, some information that retains the original structure of the data is kept, for example, xlsb for binary Excel data.
Ideally, when decoding the data back into its original format, we want some form of "typed conversion", where we can access the original data in the format it was originally stored.
Let's see how to do that using the Python library pandas, motivated by a short real-world example.

An example to set the stage

Let's assume that we have real-world data containing information about the changes in acceleration on the crest of a dam over time, measured by accelerometers installed on the crest itself. This data is recorded in an Excel file containing two columns: t, representing the time in seconds, and a (mg) the corresponding acceleration value in micro-g.
Now, let's suppose, for monitoring purposes and displaying chart data, this file is encoded into binary and stored in a MongoDB DB as a binary field, so, when looking inside the database documents, for example, via a managed Atlas database, we get:

Note how it is stored in a binary format in the DB. Document databases are well-suited for storing this style of unstructured data, which is why we show a NoSQL database here, but, a SQL database would work just as fine.

So, now, we have a field in a database that is nothing more than a plain stream of binary data. Let's see how to get a typed decoding from it, so, we can effectively rebuild our original excel data to be used by any client app.

Using pandas+BytesIO to get a typed conversion into the original excel format

Now that we have our binary data, we need to consume it at the client side. In order to do it, we can leverage a combination of two distinct packages: BytesIO, a standard Python package that can parse a stream of bytes using a specific encoding, and pandas, which, quoting the official website, " is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.". Note that the read_excel built-in function from pandas requires installing an additional dependency: openpyxl.

The trick is to use composition of the function calls, by nesting them inside one another: first we process the binary stream with BytesIO and then, using that parsed stream, we can pass it into pandas, and, in particular, to the read_excel function.

Let's assume that in our mongoDB database, we have the binary file stored in a list called dataFiles and we want to parse the first file in the list. Assuming mongoClient is a configured and ready-to-use client, the chaining code would look like:



import pandas as pd

(...)

excel_data = pd.read_excel(io.BytesIO(mongoClient['dataFiles'][0]))

Now, inside the variable excel_data we have an array of Nx2 entries, where N is total number of rows in the excel file, which, in this example is 4000, so, when we print the variable to the console, here is the output:



         t    a (mg)
0      0.00  0.000358
1      0.02 -0.000466
2      0.04 -0.000181
3      0.06  0.000697
4      0.08  0.000618
     ...       ...
3996  79.92 -0.001205
3997  79.94  0.000120
3998  79.96 -0.000061
3999  79.98 -0.000022
4000  80.00  0.000142

exactly as the original excel file format, decoded from the binary format correctly!

Now, if we want to have access to a column, we can actually do it in a typed way, so, printing excel_data['t'][0] will give us back the value 0.00 as a 64 bit float value, typed, as we wanted.

Conclusion

Using a combination of standard library and open-source libraries, we can see how simple it is to convert binary data to an Excel format and process and operate on it.

The main trick is to use functional composition of different libraries in order to have the data flowing in between them in the required compatible formats, and, finally, being outputted in a typed format (can be either numeric or text, dependending on the data that was in the Excel). Then, the resulting 2D array operates like an hashmap where the keys are the headers from the various Excel columns.

Go broader to go further - leverage your learning chances

Bruno Oliveira — Wed, 20 Oct 2021 15:25:02 +0000

Introduction

This will be a short piece about how to grow in your career from a perspective of someone who has been around the industry for about 6 years. I believe this is neither too long nor too short, but, instead, just right, to be able to throw in some advice about what has been working for me recently in terms of growth. Obviously, as with all these types of blog posts, everyone's circumstances will be different, companies will be different and the combination of previous work experience that makes your own skillset a unique one, will also be different. So, what has worked for me, may not work for someone else. Nonetheless, I believe there is value in exploring the idea of going broader to go further. This can have several different meanings, depending on whether it's applied to your $DAYJOB or to some side-projects or ideas you might have, but, I will keep the list uniform for both contexts here.

Practice writing

Writing is at the core of what we do as developers: we write code, tests, tickets, outage reports, etc. Writing is ubiquitous in development work in many different forms.
However, what I mean here by practicing to write is a bit more specific. I mean, really write.

Nowadays, most companies do have some form of external blogging or even newsletters to share their updates with interested people, like, current customers, current/former/future employees or other developers who have an interest in learning about some particularly interesting technical solution for a problem. If your company doesn't do it, see if that's an area of opportunity you can explore.

This is the kind of thing that is usually non-technical, since, it mostly involves preparing and writing text, usually together with some marketing or copyrighting team, which is a great way for you to see different aspects of the large organization you are a part of.

When doing this for your own side-projects or explorations, it's a lot easier to produce much more written content, just like the current collection of these blog posts, for example. They can be more free-format and you can write about what you like. The important takeaway is that investing time in writing at a professional context can bring many benefits: visibility, improves overall cohesiveness of the codebase, facilitates onboarding new colleagues, and keeps a detailed record of architectural decisions, documentation, etc, which is invaluable in the long term.

Do something new when possible

It's easy to fall into a trap of only doing your "routine work". Same challenges, same pre-packaged solutions, time and time again. This is probably fine for a while, but, in the long run, it can become a bottleneck for how you perceive your own impact inside your company in terms of value delivered, as well as how much you can grow within a "familiar setting". So, when given the opportunity, try and "shake things up" a bit. This, again, can mean anything in different contexts: switch teams internally, give a talk about a topic you like, investigate that platform team bug to be familiar with the dark corners of your codebase, etc. These opportunities usually arise as a mix of things that are already in front of you and just need to be done, or, as things that you can create for yourself by being proactive and "championing" it to your team, department, company, etc.
It's both easy and hard to do, since it depends on your own style, the current workload you have, and other factors, but, it's a good thing to strive for.

Look into devOps, if you code, code, if you devOps - in other words : containers

Automation, batch processing, jobs, queues, pipelines, terraform, containers, etc, etc, etc.

The current landscape is combining code and infrastructure more and more, in ways not thought of before. This is great, because, its fertile grounds for innovation.
As a developer, you can now have a say in how to manage your deployments, your pipelines and even your own code, to better fit existing infrastructure, and, as a devOps person, you can now write code to automate a lot of stuff that was more on the configuration side before.

Containers make it look almost seamless in the way they "merge" both worlds so well, and, I believe that they have been the best innovation in the industry for product-based development companies within the last 5/6 year period since starting my career. Writing integration tests is now a breeze thanks to containers. Testing new language releases, handling deployments with tools like docker swarm or k8s is more and more becoming the norm.
If you really want to explore new possibilities for your own workflows, side-projects or simply to help move your company forward, I can safely recommend learning Docker as well as possible, and, delve a little bit into Kubernetes, as these tools are becoming pivotal for many, many companies. So it will pay off in both the short and the long term to learn them well.

Subscribe to dedicated newsletters or read blogs you like

I think this has helped be increasing my impact tremendously, in small ways that pay off in the long run: by reading and consuming information from newsletters and blog posts on topics like code quality, leadership, management and career paths, I've come to appreciate that there are many things that YOU as an individual contributor (IC) can do, in order to help your team and company grow: writing is very important, and, simultaneously, keeping an eye out for small process improvements, is a great combination that, when rallied around the right people, can really bring your impact to the next level through your interpersonal and soft skills, which, can be just as impactful as technical skills.

Stay on top of what's new in the industry

Somewhat linked to point 4 above, but, as a very concrete example: You're using Java 11 at work? Great. Java 17 was just released. What's a better weekend project than simply trying it out for yourself?
Even better, if you can scope down and/or recreate smaller snippets of code that would resemble a codebase you have to deal with at your job, you might discover a lot of benefits in thinking how that new release of your favourite language or framework would definitely improve areas A, B and C of the codebase. This is valid not only for very tangible things, like new languages or frameworks, but, also, for smaller updates on other productivity tools, like GitHub, Jira, Slack, etc. If you invest some time in learning what is new on these tools, you will often be surprised how many new tiny things can help you or your colleagues in a very tiny, almost too simple way. But, when you start compounding these small things over time, you can get some really nice benefits that you'd might not have even hoped for initially. So, if you can and are willing to, stay on top of your game. It's easier than you think when doing it for a living.

Conclusion

These are just five simple things that I've found myself doing, more and more, especially over the last year and a half, and, I can say that its combination has helped me grow as an engineer tremendously fast. Much faster than if I had simply stuck to do my normal 9-to-5 job and attempted to simply learn as I did things.
It's also something that does feel very natural to do, because it uses the "compound interest effect": I do a tiny bit every day, and try out a tiny thing here and there... and... After all these years so far, when I look back, I'm definitely reaping the benefits of doing these smaller things over time! Hopefully, this can inspire you to look into broader horizons as a way to power your learning chances!

Building Kubernetes-first apps: yay or nay?

Bruno Oliveira — Sun, 25 Jul 2021 10:08:41 +0000

Introduction

There has been a large number of posts recently about Kubernetes and a particular one from the team at Stackoverflow actually advocates about building on kubernetes from day one.
Their post is very well-written down and provides good arguments on why it can pay off to keep it top of the mind when building things. I will offer my perspective on this as a developer in a company that is adopting kubernetes and as a maintainer of a personal side project where I use kubernetes to manage my backend code deployments.

Are you ready for Kubernetes?

Most projects I've worked with, both professionally and personally, in the last 5 years, have all leveraged Docker in some shape or form: Some used it for actual deployments in very manual, simple ways: think _rsync_ing files into a container after a compile step and then starting it up. Others used it for several stages of the SDLC from testing all the way to production, and, others yet, simply used it as a way to "package your code to make it ready to run somewhere else the world can see" - yes, this last piece is my side project :)

If you are using Docker, and, equally important, if you are structuring your workflows and code organization around your usage of Docker, for e.g, by building modular code in a microservices style, or using Docker to run pipelines with integration tests that give you quality assurance, or simply, if you are keeping your code dockerized and storing the images in a container registry, then, you are most likely ready for Kubernetes.

Leveraging your readiness

Okay, cool, so you're already using Docker, you know a bit about Kubernetes, what it promises to do and how many companies are shifting towards adopting it. Time to leverage your readiness and really jump into it. But how?

Well, as defined in the stackoverflow post above, the rise of managed Kubernetes services, like Google Kubernetes Engine, EKS, etc, make it easier than ever to actually have a cluster ready to be used. Or do they?
In my personal experience, these services are really invaluable in allowing me as a developer, whether I'm working on my own side project, or my company, to experience Kubernetes usage from a practical perspective, i.e the perspective of a developer who wants to simply deploy code into a cluster and make it run there. But, there are a lot of nitty-gritty details that still need to be resolved whether with the help of platform teams or devOps folks or simply tight collaboration with your senior colleagues. Some examples of things (technical and non-technical) that can be really tricky and need handling:

permissions in AWS to manage/create, deploy to, or configure an EKS cluster. These are hard to hand out and create, and hard to get right from the get go. There has to be a designated person as the ´cluster owner´ that can facilitate such tasks;
configuring things like Ingresses, Load Balancers and memory and CPU usage limits. Yes, these things can be used out-of-the-box in these managed services, but, to really adapt them to your own needs, like, for your company's proxy rules, several docker images, metrics collection, etc, are the hard things nobody talks about it, and these take time and effort;
manage your databases properly and keep your database out of the cluster and out of containers. Databases should be managed separately as they are of a different nature than that of your application code. It's not a pick all, mix all kind of approach. This requires some infra work to get right though!
bring people into it. Yep, these days it's surprisingly tricky to navigate an environment with the idea to convince people to "join you". You need to do real devRel advocate work inside your own team (first), and collaborating teams (after) if you want the adoption to succeed. It needs to really be a team effort;

How to empower people to try it out

Part of the success when leveraging the readiness to go into Kubernetes is in ensuring that everyone is on the same page. For this, there are several online workshops from many different cloud vendors that can ensure that everyone in your team is looking at the same concepts with a very similar level of exposure. This, in turn, will bring more confidence to everyone in the team to try these things out and see what the hype is all about!

You can even push the envelope when suitable and, well, remember that cool side project where you used Kubernetes on your own for the first time? Maybe talk about it at your work! You may be surprised by the positive reception and feedback that these initiatives can have. Sometimes, a nudge in the right direction is all it takes!

You, as an individual

For you as an IC, the best way to really learn, is to commit to a simpler managed Kubernetes service (eg. DigitalOcean) and just...build something! A small project for you to get exposure to some of the basic things that are available to you, while, at the same time being exposed to some "real world" complexity that comes with you having to manage the pods, namespaces, ingresses, etc, all on your own. It's exactly what strikes the right balance between complexity and learning, in my opinion. From there you can keep building more knowledge, contribute to real things inside your work and keep ramping up!

Conclusion

Hopefully, after reading this post you will feel more ready to start looking into bringing Kubernetes into your organization or to simply start using it yourself! It's an exciting time to be on top of the wave of Kubernetes and it's not THAT hard to at least make a jumpstart for its adoption and to see the benefits! Keep sailing!!

Practical Java 16 - Using Jackson to serialize Records

Bruno Oliveira — Sat, 24 Apr 2021 14:21:01 +0000

Introduction

With JDK 16 being in General Availability and JDK 17 already in early access, I think it's finally time to explore what is arguably one of the coolest features offered by the "bleeding edge" Java: records, and how to work with them from a practical perspective.

One of the most common workflows for modern applications, relies on serializing what is commonly referred to as a "Data Transfer Object", shortened to DTO, that is an object used to represent a domain concept that we want to serialize to a common format that can be consumed by clients, usually JSON.

A common library to do this in the Java stack, is called Jackson that offers core functionalities and databinding capabilities to serialize Java classes to JSON and to deserialize from JSON into Java classes.

We will see how Jackson works and how records are so much more compact and easier to use.

Current status and usage before Java 16 and Jackson 2.12.x

Currently, using Java < 16 and Jackson, since records are not supported, the only way we have to serialize a Java object into JSON, is to use the Jackson object mapper, as follows:

Let's create a simple PersonDTO class which has the following information:

public class PersonDTO {
    private final String name;
    private final int age;
    private final double weight;
    @JsonSerialize(using = LocalDateSerializer.class)
    private final LocalDate collegeApplicationDate;

    public PersonDTO(String name, int age, double weight,
        @JsonSerialize(using = LocalDateSerializer.class) LocalDate collegeApplicationDate) {
        this.name = name;
        this.age = age;
        this.weight = weight;
        this.collegeApplicationDate = collegeApplicationDate;
    }

    public String getName() {
        return name;
    }

    public int getAge() {
        return age;
    }

    public double getWeight() {
        return weight;
    }

    @JsonSerialize(using = LocalDateSerializer.class)
    public LocalDate getCollegeApplicationDate() {
        return collegeApplicationDate;
    }

As we can see, this is quite verbose, and arguably, most of this boilerplate could be avoided, and the information conveyed by the class would be exactly the same: it's a person with some personal information and an application to a college date, nothing else.

The constructor, the getters and the attributes, add a lot of clutter, but, it's the only way to have a DTO to be serialized by Jackson, as the getters are required so that the library can write them to JSON internally, if we are below Java 16.

Let's use the below service as a kind of placeholder to exercise Jackson:

public class PersonListingService {

    public PersonListingService() {
    }

    public List<PersonDTO> getPersonList() {
        return Arrays.asList(
            new PersonDTO("John",30,60.7, LocalDate.now()),
            new PersonDTO("Shaw", 33, 70.5, LocalDate.now().minusDays(20L)),
            new PersonDTO("Harold Finch", 50, 70, null));
    }
}

We can write a quick test and verify that this works as expected, using Jackson 2.11.4, that comes bundled by default with Spring Initializr in IntelliJ:

 @Test
    void dto_list_can_be_serialized_via_jackson() throws JsonProcessingException {
        ObjectMapper mapper = new ObjectMapper();
        mapper.setDateFormat(new SimpleDateFormat("yyyy-MM-dd"));

        String serializedList = mapper.writeValueAsString(personListingService.getPersonList());

        Assert.assertEquals("[{\"name\":\"John\",\"age\":30,\"weight\":60.7,\"collegeApplicationDate\":\"2021-04-24\"}"
                + ",{\"name\":\"Shaw\",\"age\":33,\"weight\":70.5,\"collegeApplicationDate\":\"2021-04-04\"},{\"name\":\"Harold "
                + "Finch\",\"age\":50,\"weight\":70.0,\"collegeApplicationDate\":null}]", serializedList);
}

This works in Java < 16 and with Jackson 2.11.4, because, the getters in our DTO are used internally by Jackson to map the fields that need to be translated to the JSON: it knows to write an age field because our DTO has a getAge method, and this correspondence is done based on the getters and attributes names, by default.

Let's see now how to make records work with the Jackson version that comes bundled with the Spring Initializr.

Serializing a record using Jackson 2.11.4

We can now "convert" our PersonDTO into a Record, very easily:

public record PersonDTO(String name, int age, double weight,
                        LocalDate collegeApplicationDate) {
}

A Java Record can be thought of as a data class that has specific fields, and can be created as a normal class, using the standard constructor based on the attributes defined above. This means that our original service works as it is declared above, no changes needed.

However, if we try to serialize this as it is above, our test will fail:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class com.example.demo.dto.PersonDTO and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: java.util.Arrays$ArrayList[0])

Let's try to add a @JsonSerialize annotation above the record:

@JsonSerialize
public record PersonDTO(String name, int age, double weight,
                        LocalDate collegeApplicationDate) {
}

and we get:

org.junit.ComparisonFailure: expected:<[{["name":"John","age":30,"weight":60.7,"collegeApplicationDate":"2021-04-24"},{"name":"Shaw","age":33,"weight":70.5,"collegeApplicationDate":"2021-04-04"},{"name":"Harold Finch","age":50,"weight":70.0,"collegeApplicationDate":null]}]> but was:<[{[},{},{]}]>
Expected :[{"name":"John","age":30,"weight":60.7,"collegeApplicationDate":"2021-04-24"},{"name":"Shaw","age":33,"weight":70.5,"collegeApplicationDate":"2021-04-04"},{"name":"Harold Finch","age":50,"weight":70.0,"collegeApplicationDate":null}]
Actual   :[{},{},{}]

So now, we did manage to "serialize" something, but, we get an empty array, meaning that each individual record couldn't be serialized correctly.

Without any real "insight" on how records truly work under the hood, it can be hard to debug this, but, it's worth having a deeper look.

"Debugging" records serialization with "old" Jackson

IntelliJ offers us the possibility to convert a record to a class, so, let's do it and see exactly how the JVM "sees" a record as a true data class under the hood:

@JsonSerialize
public final class PersonDTO {
    private final String name;
    private final int age;
    private final double weight;
    @JsonSerialize(using = LocalDateSerializer.class)
    private final LocalDate collegeApplicationDate;

    public PersonDTO(String name, int age, double weight,
        @JsonSerialize(using = LocalDateSerializer.class) LocalDate collegeApplicationDate) {
        this.name = name;
        this.age = age;
        this.weight = weight;
        this.collegeApplicationDate = collegeApplicationDate;
    }

    public String name() {
        return name;
    }

    public int age() {
        return age;
    }

    public double weight() {
        return weight;
    }

    @JsonSerialize(using = LocalDateSerializer.class)
    public LocalDate collegeApplicationDate() {
        return collegeApplicationDate;
    }

    @Override
    public boolean equals(Object obj) {
        if (obj == this)
            return true;
        if (obj == null || obj.getClass() != this.getClass())
            return false;
        var that = (PersonDTO)obj;
        return Objects.equals(this.name, that.name) &&
            this.age == that.age &&
            Double.doubleToLongBits(this.weight) == Double.doubleToLongBits(that.weight) &&
            Objects.equals(this.collegeApplicationDate, that.collegeApplicationDate);
    }

    @Override
    public int hashCode() {
        return Objects.hash(name, age, weight, collegeApplicationDate);
    }

    @Override
    public String toString() {
        return "PersonDTO[" +
            "name=" + name + ", " +
            "age=" + age + ", " +
            "weight=" + weight + ", " +
            "collegeApplicationDate=" + collegeApplicationDate + ']';
    }
}

So a record is nothing more than a "final class" with final fields, and an auto-generated hashcode and equals methods. Interesting.

It's especially interesting from a debugging perspective, because, as we noted earlier, Jackson 2.11.4 requires the getters to be defined explicitely in order to be able to serialize the class to JSON, and, from this representation, it seems like the field names are being used directly to generate the getters, which opens up the road to the next logical step.....

.....

@JsonSerialize
public record PersonDTO(String getName, int getAge, double getWeight,
                        @JsonSerialize(using = LocalDateSerializer.class) LocalDate collegeApplicationDate) {
}

Don't do this at home!! :D

We can actually trick Jackson's reflection mechanism as well as the JVM record conversion into generating "getters" in the underlying data class, which CAN be picked up by Jackson...

Expected :[{"name":"John","age":30,"weight":60.7,"collegeApplicationDate":"2021-04-24"},{"name":"Shaw","age":33,"weight":70.5,"collegeApplicationDate":"2021-04-04"},{"name":"Harold Finch","age":50,"weight":70.0,"collegeApplicationDate":null}]
Actual   :[{"collegeApplicationDate":"2021-04-24","name":"John","age":30,"weight":60.7},{"collegeApplicationDate":"2021-04-04","name":"Shaw","age":33,"weight":70.5},{"collegeApplicationDate":null,"name":"Harold Finch","age":50,"weight":70.0}]

While the fields of the objects are out of order, we see that we do have the data there, and actually it works because, the values in the fields are exactly the same. We left the custom serializer to handle the date, but, using the "artificial getter" actually was a funny way to leverage record's internal construction structure.

The correct way is to use Jackson's annotation AutoDetect:

@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
@JsonSerialize
public record PersonDTO(String name, int age, double weight,
                        @JsonSerialize(using = LocalDateSerializer.class) LocalDate collegeApplicationDate) {
}

this will let Jackson autodetect the fields based on the record's attributes and the test will pass, since the restriction of the getter field will be lifted.

Since records generate the getters directly from the attributes of the record, this is a good way to bypass it, that is perfectly valid and allows us to use records today with Jackson 2.11.4.

How to do it the right way

By upgrading Jackson to it's latest release 2.12.3:

  <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.12.3</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.12.3</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-annotations -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.12.3</version>
        </dependency>

We can make use of the great interoperability between Jackson and records, thanks to the added support for record serialization (more info here ) which means that simply declaring a record in its simplest form:

public record PersonDTO(String name, int age, double weight,
                        @JsonSerialize(using = LocalDateSerializer.class) LocalDate collegeApplicationDate) {
}

will work, and the record will be able to be serialized correctly, leveraging the local date serializer as well.

Conclusion

As Java keeps improving and maturing, some of its features are extremely useful and versatile, and it's always nice to explore them in some detail to understand its nuances and how it works under the hood. Our example of leveraging the knowledge of Jackson together with the way JVM generates a record is a good example of that.

With the right configuration, records can work almost hassle free with Jackson 2.11.4 and it's as seamlessly integrated as normal workflows of today in Jackson's latest releases. Are you using records already? :)

A reflection on modern software development

Bruno Oliveira — Sun, 21 Feb 2021 00:12:06 +0000

Introduction

Writing software has changed a lot in recent years, and it’s now more complex than ever before. The complexity of the businesses relying on software, ingrains a natural complexity into the domains for which we, as developers, have to write software for.

My career in the field started only a mere 5 years ago, but, I’ve already seen a few different businesses and different ways of working and managing teams to be able to compile this post offering some advice that can help you manage some of the inherent complexity while keeping the accidental complexity at a minimum.

Accidental and inherent complexity

The inherent complexity is the complexity that you have to face because of the nature of the business you are writing software for. If you are writing a guidance system for an airplane or devising a complex algorithm to help you synthesize a new drug, then, the complexity of how a guidance system works, how a plane orients itself, how it manages lift, drag, responds to pilot commands, or a new drug is deemed efficient or not.... All of these facts about your domain, in one form or another, will have to end up making it into the code. This is the inevitable piece of the puzzle, there’s no escaping it.

The art of writing good software is in how we can embed this necessary complexity into the “accidental one”: the language and/or tech stack we are using to encode these real-world facts into code that can deliver the real value for the business. Anyone would agree that writing a guidance system for an airplane in Assembly is a terrible idea: we have better tools. Yes, better tools like...Python. Except that Python is a terrible idea for a high-performance, real-time system like a guidance system. Any programmer reading this can relate.. It’s just not fast enough, not performant or memory efficient enough to be running in constrained environments. C or C++ or Rust are better choices. Much faster execution cycles, the generated native code can be fully optimized to a target architecture chip, full control over memory, etc.

Knowing that we need to always choose the right tool for the job is a key aspect of managing accidental complexity.

The rest of this post will focus on how we can write good code, following best practices and using modern technologies from the perspective of non mission-critical domains. Google’s search engine, a hotel reservation system, tools to analyze code, CAD tools, etc. are all non mission critical domains. Failures in these systems can have consequences and are very serious, but not as serious as a self-driving car brakes failing or an airplane falling off the air…

Taming complexity: a toolset to be efficient in the modern software development age

I will approach this section from the perspective of someone who works with Java, Springboot, Git and uses Docker and Kubernetes to manage and provision infrastructure, but the underlying principles will hopefully be general enough that anyone will be able to get something out of this post.

Teams and processes

Writing software professionally, for a very large amount of developers, is at its core a collaborative activity. This means that you don’t work alone, and you need to collaborate with other developers and report to managers, product owners, follow the Agile methodology (if your company uses it) and most importantly, you need to communicate and agree on how to work and on how to prioritize what to build next.

This is really the crux of the matter: agreement, low barriers at entry, and managing friction and conflicts between teams and individuals is the most important thing. It’s independent of any methodology you or your company decide to adopt but, it is the single most important aspect that you need to get right to enable yourself and your colleagues to work efficiently. All of the other things, be they methodologies, processes, certifications, whatever, only exist to enable and foster transparency and clarity of requirements between team members and other relevant stakeholders. Remember: nobody cares about story points on a ticket. People kind of feel like they need it because it is something that you can measure. But, focus instead on what you CAN’T measure: your grit, your commitment, your will to do good by your colleagues, to keep improving the product you work on, to take pride in a job well done. If you focus on these things, you end up embracing your work in a different light that can effectively let you work under ANY company using ANY type of methodology. Because, at the heart of all of it are people. And, fostering good relationships between teams and peers is the most important thing when it comes to working as a team. The rest will naturally follow and fall in place. Be kind, attentive, work hard, listen, express your points of view and everyone will benefit. Teams are not defined by their governing methodologies, they are defined by people. This is important.

Ride the devOps wave

There is no such thing as writing code in isolation anymore. Continuous Integration and Continuous Delivery are key indicators of successful development teams and businesses in today’s world. Any merge to the master branch needs to be successfully released to production with ZERO manual intervention. Build reproducibility is no longer a luxury or a nice to have. It’s key to the efficiency of a development team.

Docker and Kubernetes were released, respectively, 7 and 6 years ago. A developer who started working professionally in 2012, SAW the way build reproducibility changed. He or she saw how this could shape the way entire teams work and manage their code. It was a revolution.
However, the technology is complex and it requires effort from the developers to learn it and leverage it in the best possible way. I like to think of devOps as “developers assuming responsibility for Operations”. This is critical, and, it’s one of the best investments you can make today that will keep paying itself off for the future to come.
Learn the basics of Docker and Kubernetes, and leverage it to be efficient and help your team be efficient as well. There are a plethora of aspects that you can approach the devOps wave for yourself and for your company:

improve local build reproducibility by putting some of your services into a private container registry for your team or company. Backend services can be dockerized so that a new developer joining the team can run an entire suite of integration tests with a click of a button.
Test data inside SQL databases can be dockerized using a mounted volume that can enable a script to load some static test data into a container. Your front-end team can leverage this to run visual tests, to test changes at specific levels, etc.
Create “templated” docker compose files, where you can add certain services, for example, from different branches, or for example different databases, like managed replicas created by your devOps team, or clusters from Google Cloud Platform, you name it. Docker provides you with the flexibility, safety and isolation your developers need to be as productive as possible locally. Embrace it as much as you can.
Kickstarting Kubernetes usage at your company? Look into branch deployments. Kubernetes can be used as a great tool to enable developers to spin up a complete “branch environment”, that behaves like the real production thing, but contains your surgical changes made in your branch. This is invaluable. “Will this be performant enough?”, “Does changing this endpoint mes up the front-end?” Answering these questions by enabling a full environment to be spun up with a git push is a real enabler. It’s hard to get right and takes effort, but, it’s a great way to be at the forefront of current development practices and very useful for developers. Gitlab offers Kubernetes integration, and Amazon Web Services has kubernetes clusters available, and has backed tools like cdk8s, to enable a declarative approach where we can use Java and the Builder pattern to programmatically generate yaml files that can be deployed on a running k8s cluster. Possibilities are endless.

Docker and Kubernetes tie closely with devOps in the sense that they enable developers to manage and provision resources and environments in a way that was typically reserved to “the devOps guys” a few years ago.

Writing good code

Writing good code is important, because when you’re working under tight deadlines, or when requirements shift, your code needs to be able to cope just as easily. Code needs to be easy to adapt, change or remove, and it needs to be concise and simple. Obviously, writing such code is hard and it’s influenced heavily by the technology your company is using and also by your level of familiarity of the language and the type of architecture you’re using. However, there are some guidelines that can help you shape your code so that it can remain flexible and maintainable even in the face of constantly changing requirements. This will be an incomplete, biased list, but, it should give you important insights:

Keep your services as generic for your domain as possible. If you feel that you have several areas in the code where you need to apply the same logic more than a couple of times, seek for opportunities to extract that logic into a separate, reusable service that can both increase code clarity and reduce duplication.
State and state propagation are the source of numerous bugs. If things happen over which you have no control, isolate these things proactively, and ensure that state does not leak any further than it has to. Data processing, like reading from a database, is always a stateful action, and we can’t always make assumptions that are always valid about our data and its structure: “will this field never be null?”, “Can this list be empty here?”, etc.
What I believe to be a very suitable approach, one that indirectly frameworks like Springboot guide you towards, is to leverage the concept of “Onion Architecture”. Borrowed from functional programming, this architecture, as the name states, is composed of layers, where the outer layers are the stateful ones, and where information flows from the outer to the inner layers. Isolate the data fetching and processing at a certain layer (for e.g. Spring’s repository abstraction over the database you’re using) and ensure that no information can flow from that layer to layers that are directly responsible for the logic of your business domain, like sending responses to a client.
Keep your classes and methods short, and use meaningful class names that map directly to the concepts of your particular business, use them and leverage these as first-class citizens in your code. Concept mapping builds understanding.
Try and structure the areas that handle the business logic as data transformation pipelines. Functional Programming abstractions like data enrichment, mapping, transformation, filtering, grouping and sorting, usually make handling the data all the way from the data layer right to the area where it is needed, very smooth work. Convert from database entities at the outer layer into immutable representations for your domain at the inner layers, and handle those with the functional programming tools that are best fit to do the job.
If a method or class is becoming too long, look for ways to refine your logic that allow you to compose something from smaller parts, that are hopefully, nice candidates for reuse, especially because many domain concepts are usually closely related together.
Leverage the tools existing in your ecosystem to their fullest potential: Java streams, the Function interface, Optionals and the object-oriented paradigm can help you shape your code in a certain structured way that can grow organically.
Springboot can be very powerful when we can leverage its Aspect-Oriented programming side, in order to design and build our own custom annotations and validators, literally enhancing the existing set of annotations with additional ones that are suited to our domain.
Jackson can handle serialization and deserialization very efficiently out of the box and it’s a great abstraction that we can leverage to take away most of the plumbing involved in pushing data across the network.
Write your code in this way, and you will see it lends itself well to tests. Write unit tests AND integration tests. Both types are important and they can test different layers of your architecture.
The closer you move to the “outer shell” of the onion, where things can get more stateful (like calling an endpoint or interacting with the database) the harder it is to test, you may need test data, mocks, you usually also need a whole “test application context”. This happens because when dealing with state, the number of moving pieces increases and that complexity gets reflected in the tests as well.
As you move to less stateful layers, like certain services that leverage the “data pipeline” style of code, you will see that simple, stand-alone unit tests are usually enough and that’s because usually those services are simple data transformation classes, with very few to no external dependencies, so the set of testing cases is reduced a lot. Even more when the classes and methods are small in both size and responsibilities.

With these guidelines in mind, hopefully you can strive for a design that is easy to maintain, extend and test.

Conclusion

Modern software development fuses together code and operations as never seen before, and that can bring many challenges to teams that have to adapt to new ways of working.
Leveraging simple principles across the board, from your teammates to the responsibilities of your methods, you can thrive in this modern day and age and enjoy doing it!
Strive to always keep learning, get involved in all the aspects of the software development lifecycle, stay up to date on current practices and you will improve your skills and inspire others!

Testing Amazon S3 using s3Ninja

Bruno Oliveira — Tue, 22 Dec 2020 18:51:04 +0000

Introduction

AWS is now a common service being used in many web applications, being used for file uploads, storage, container registries, hosting, as a container orchestrator platform, among many other things.

As its usage grows, developers also have increasingly higher needs of testing the integration of such complex services with their applications, and this can be no easy task, as these services require a lot of configuration and are very complex on its own.

Thanks to the increase of containerization and popularity of docker, we are now also seeing a trend where the community is providing and maintaining docker images that facilitate the work of developers needing to test the integration of their application services with AWS services.

We will look at one of the most popular images, scireum/s3-ninja, how we can configure it, and how to setup a simple IT for one of the services in our application that depends on Amazon S3.

Basics of S3Ninja

Upon starting the docker image, here is what we see:

As stated in the image above:

S3 ninja emulates the S3 API for development and testing purposes.
It is however not intended as production system as it neither provides scalability nor replication or proper security.

Accessing the running container in a web browser, and if we navigate to, for instance, http://localhost:32782/ui/api we can see the supported API and what type of methods we can test.

While the container is running, we can configure, programatically, from the Java side, a "mock AmazonS3 client", by exposing the URL of the running container, and we can then use this mock client to perform our integration testing.

Configuring the mock S3 ninja

To configure the mock client, we need to spin up the docker container that has the S3Ninja image, and then use it to configure the client:



@Testcontainers
@ExtendWith({SpringExtension.class, MockitoExtension.class})
@SpringBootTest
class TestingAServiceThatRequiresUsingAmazonS3Test {

//Container configuration using the Testcontainers dependency to manage docker containers from test contexts
    @Container
    private static GenericContainer s3Ninja = new GenericContainer<>("scireum/s3ninja").withExposedPorts(9444);

(...)

private void setupAmazonS3ninja(String bucketName) {
        final AmazonS3 httpAmazonS3 = AmazonS3ClientBuilder.standard()
            .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(
                "http://" + s3Ninja.getContainerIpAddress() + ":" + s3Ninja.getMappedPort(9444) + "/",
                Region.EU_North_1.getFirstRegionId()
            )).build();

//Inject s3Ninja into the service for testing purposes  
  someServiceThatRequiresUsingS3.setupAmazonS3client(httpAmazonS3);
    }

The basic scaffold above leverages the Testcontainers maven dependency that automatically manages docker containers for use from a test context, and below that, we call a public method from our Spring managed service whose sole purpose is to "inject" the mock client into the service.

Here it's passed via an explicit method, and not using for example, constructor injection, since the mock S3 client is a regular dependency, not managed by Spring. By using a method like this, we can control between using real clients or mocked ones from a test context, while still retaining the flexibility to parameterize it for production usage.

With this simple setup, we can now write an actual test.

Writing a test

Leveraging the above scaffold, writing a test becomes very easy:



@Test
    void correct_setup_returns_good_response_from_service() {
        setupAmazonS3ninja("mytestBucketName");

        Optional<SomeResponseDTO>response = someServiceThatRequiresUsingS3.callAmethodThatUsesS3ClientInternally();

        assertTrue(response.isPresent());
    }

As we can see, once we setup the mock S3 client, from the running docker container, we can now effectively unit test our service method against a fixed s3 client that allows us to test exactly the business logic we need.

Conclusion

This post describes one possible approach to have a basic configuration to test services that rely on Amazon S3, by using S3 Ninja.

Thanks for reading!

Architecturing Spring services

Bruno Oliveira — Wed, 07 Oct 2020 11:17:57 +0000

[disclaimer: this post will go in depth into many aspects of one possible type of architecture for microservices, and will use Java and SpringBoot to showcase examples. It will not show a full and complete working SpringBoot setup and it will assume that basic principles can be applied to any technology stack. The foundational ideas are what matters most from an architecture point of view]

Introduction

Microservices are best described as an architectural style that structures an application as a collection of services that have the following characteristics:

Loosely coupled;
Highly maintainable and easy to test;
Owned by a small team;
Structured around business concepts;
Allow for structured and independent deployment;

It's an architectural style that has gained traction during recent years, with many companies sharing their experiences with this architectural pattern (see Uber for example) on their technical blogs, and many whitepapers praising the advantages related to scalability, ease of development, and adaptability to changing requirements when compared to the traditional monolith approach.

This post will attempt to describe one real-world example on how to architecture microservices with Java and SpringBoot, while taking detours where needed, either to explain particular Spring concepts or abstractions, or to frame how the example architecture matches the ideas that compose the pattern itself. This is important because it gives readers a real frame of reference as to where a real-world architecture stands when compared to the idealized, theoretical foundations of the pattern.

The differences between theory and practice are where the fun is at and understanding how some basic principles present in frameworks like Spring can actually facilitate certain types of development can pay off when looking at things from a different lens. Before we begin, we can first look at why it can be a good idea to use microservices.

Framing microservices in the modern world

Everything is fast-paced and ever evolving nowadays, and code is no exception to that. In fact, according to a post at ArsTechnica, developers claim managing 100x more code today than they did in 2010. That's a huge increase. And the tendency is to keep growing!
With more and more companies jumping on the digital transformation bandwagon, we now see developers and developer job positions at companies whose primary focus of business is not even technical: retail, insurance and even food business are now contributing to the code footprint of today.

This means that the original approach of coding in a monolith no longer scales well enough to meet the demands, and speed and innovation have become key factors for success, and here is where microservices architecture can provide that much needed competitive edge. Let's now delve into the differences between microservices and a traditional monolith approach.

Differences between a monolith and a microservices approach

When it comes to identify the differences between these two patterns, it's important to notice that these architecture patterns appeared at different times, and solved different problems for managing complexity.

Using a monolith-style approach means that all of the codebase exists and lives and grows as a single, deployable, unified unit.
There can be a clean architecture internally, each module can be well structured and written, but, the whole application and codebase can only exist as a single deployable unit.

Let's look at a shopping cart application, for example.

Monolith lens

There will be obvious pieces of logic that can be seen as orthogonal in the application: there's user management, there's a payment system, displaying lists of products, updated product stocks, the status of the shopping cart, managing the UI, etc, etc.

When developed as a monolith, we have a single system that is responsible for managing every single, orthogonal aspect of this application on its own. This implies that in a monolith scenario, if we would need to deploy a change concerning the payments system exclusively, the entire application would need to be deployed as a whole, because the several business concerns are part of a single, isolated context that can't exist as separate units.
This rapidly causes the codebase to grow to an unmaintainable state where adding new features becomes more and more costly overtime, until a breaking point is reached where the time-to-market for new features no longer outweights the time it takes to develop and maintain these additional features, effectively resulting in the death of the application.

Microservices lens

If we look at the same example as above from a new perspective, we see how there are many opportunities to structurally break the application apart in independent pieces that when looked at as a whole form the same application, but in a much more flexible way. As described above, these are some examples of some services we have to deal with:

Payments service;
User management service;
Shopping Cart service;
Product listing service;
(...)

Since each of these services is concerned with a specific context of the business logic, this means that the composition of all these services together will form our application.

The advantage of breaking down the monolith is that we can see our application as a set of independent running services:

PaymentsService, UserManagementService, ShoppingCartService, ProductListingService, ...

When using this setup, if we decide to make changes to the way the application handles payments, we can simply adapt the code and internal logic in the PaymentsService, we can redeploy only that particular service, and things will be now running on the updated service. It's safer, faster and more easily managed, since we only needed to concern ourselves with the piece of logic handling exactly the business domain that needed changing.

Going a bit on a side-note here, this style of thinking about microservices can be seen as a "domain-oriented microservice architecture", because each microservice will form a domain-bounded context, which means that each service will handle a specific part of the business domain.

It's also important to notice that microservices are very well suited to work in a CI/CD environment due to the fast feedback cycle and higher modularity and availability. Today, it's very easy to ship APIs as docker services via CI/CD pipelines and deliver code much faster than when using a monolith. This makes it almost a requirement nowadays to develop architectures using microservices where allowed and valid to do so.

Not every application can be architectured as microservices, but, if you can do it, then it really is the best approach.

Main components of our microservices code architecture

Now that we have seen the main advantages of using a microservices oriented architecture, it's time to start looking at the main components and abstractions we will leverage when building our own implementation of the theoretical microservices architecture model.

When designing a microservices architecture with SpringBoot and Java, some basic building blocks need to be in place. The first thing to discuss is how to actually expose the microservice architecture to be consumed by clients:

This will be done using a REST API that will expose dedicated endpoints that will then delegate the logic to our services (hence the name microservices), and return a JSON payload as the response. Like this, any client, be it a web app, a mobile app, etc, can just call the API we are exposing and retrieve responses in a JSON format that can then be displayed in a web page, mobile application, etc.

The REST API is the surface layer that interfaces between the clients and our microservices architecture, so now we need to detail how the actual microservice code will be structured.

Detailing the microservice code architecture

The architecture we will follow makes use of SpringBoot abstractions that translate very well into working production code. Let's see schematics of it first and we will go into more detail of each aspect as we go along:

Let's now focus on detailing the components in the image above.

Processing requests with Spring's @RestController annotation

When a request comes in from a client, the entrypoint will be what is called a "resource class". In SpringBoot, the resource class is the class that defines the endpoint URLs that will be hit when a client requests a resource.

A blueprint of such class can be as below:

@RestController
public class ShoppingCartResource {

    private final PaymentService paymentService;
    private final ProductListingService productListingService;

    public ShoppingCartResource(PaymentService paymentService,
        ProductListingService productListingService) {
        this.paymentService = paymentService;
        this.productListingService = productListingService;
    }

    @GetMapping(value = "/list-products", produces =
        MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<List<ProductDTO>> getProducts() {
        return productListingService.getAllProducts()
            .map(product -> ResponseEntity.ok().body(product))
            .orElse(ResponseEntity.notFound().build());

    @PostMapping(value = "/checkout/{cartId}", produces =
        MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<String> processPayment(@PathVariable String cartId) {
        return paymentService.performPayment(cartId);
 }

There's already a lot going on at the code architecture level, even in the resource class, so let's see how it's all wired.

The class is annotated with the @RestController annotation, which means that it will tell Spring that we will be defining endpoint methods, that can be hit by clients from the outside to retrieve data using our API.

We can see that the endpoint methods are annotated with annotations like @GetMapping and @PostMapping. This means that there is a URL mapping to a particular Http verb for a specific URL. If we look at the /list-products above, we see that it is defined as a Get mapping. This means that requests made to this URL, will be of the format:

request:
Http GET https://<the production deployment URL>/list-products

and we also declare that the response will be in JSON format. No other methods are allowed in this specific endpoint.

The reasoning for the @PostMapping endpoint would be similar.

Autowiring and injection of services in the resource class

We see above that the resource class we created has the services injected in the constructor.

In recent Spring versions, this actually does the same as having the @Autowired annotation in these classes, which means that since we are passing these services as constructor parameters to our resource class, Spring will know which services to initialize and instantiate so that everything works.

We can see that the endpoint methods, delegate all the internal logic required to the service, so, they act essentially as a "routing" layer between the client request, and the internal logic necessary to fetch the data we need to serve the client.

These are the microservices in the architectural name of "microservices": they are in essence, very small, dedicated services that do a single thing when it comes to handling a specific business domain context.

We can already see the first level of "indirection" in our architecture:

Resource classes should not know anything in terms of business logic, this should be always delegated to services, with the endpoint methods acting as "routers" between client requests and the actual data fetching/processing.

Let's now delve deeper into what constitutes a service in our architecture.

Structure of a service

As we saw before, services are the center piece of this architectural pattern, and, they are also composed of additional abstractions, as they are the components responsible for dealing with the more complex logic that will be the added value of our API.

Not only are they more complex in terms of size, complexity and logic, usually, services are also responsible for interacting with the persistence layer of our application. This means that services handle and manage accesses to the database to retrieve data and actually implement the logic we need on top of the data we need.

Services also have the characteristic of being easily composed, like pure, functional-style building blocks, we can rely on simpler services to build and extend more complex, higher-level services.
For example, suppose that on the service for listing products, we are only interested in showing in the list, the products that effectively are available in stock.

We could write all that logic as an integral part of the ProductListingService, but, that would make this service more complex to maintain, and, more importantly, the service would have two distinct responsibilities: to verify the stock for that particular product, and then prepare the data to do the listing. Instead, we can extract the logic for checking the stock into its own separate service, called for example, StockAvailabilityService that can then be reused in different contexts, since it can be useful in different contexts, like to apply promotions, or when the business grows and stock availability needs to be checked by geographical area, etc, etc. Let's look at an example of the structure of a service for listing products:

@Service
public class ProductListingService {
    private static final Log LOG = LogFactory.getLog(ProductListingService.class);

    private ProductRepository productRepository;

    public ProductUpdatingService(ProductRepository productRepository) {
        this.productRepository = productRepository;
}

public List<ProductDTO> listProducts() {
      return newArrayList(productRepository.findAll());
}

The basic idea is that, if we have a stand-alone, reusable service, then we can focus on what it offers us in terms of business capabilities and see where exactly in the landscape of the business it can be reused or composed into higher level services. This is very valuable and it's one of the main arguments in favor of microservices.

So, we have seen that services have the responsibility of handling the persistence layer and managing data manipulation to solve specific business needs. However, services also have another very important role:

Services have the responsibility of translating the data formats back and forth between the application layer and the persistence layer

We will now look more in-depth into the actual abstraction that services use to communicate with the persistence layer, and, once that view is complete, we can zoom back out and look at the different data formats and why they are useful in our context.

Introducing the Repository pattern as an abstraction over the persistence layer

As we have seen in the example above, our service relied on interfaces that we call repositories to abstract over the persistence layer, and to give us, as programmers, a higher level abstraction over the data model being used for our application, as well as some useful methods implemented out of the box.

A repository is an interface at code level, that allows us to have a set of methods provided by SpringBoot that allows us to retrieve, update, create and delete entities out of the box, and, additionally, allows us to write our own custom queries for retrieving data in more complex ways.

An example of a repository class is the following:

@Repository
public interface ProductRepository extends CrudRepository<Product, Long> {
  //this already offers methods inherited from CrudRepository: findAll(), findById(), deleteById() and updateById()
}

In addition to this, we can have custom methods driven by custom queries to retrieve data in more complex ways:

@Repository
public interface ProductRepository extends CrudRepository<Product, Long> {
      @Query("select p from Product p where p.name in (:productNames)")
      List<Product> retrieveProductsUsingTheirNames(@Param("productNames") List<String> productNames);
}

We see that with the annotation above the method name specifying the query, we can retrieve data with very fine-tuned queries that meet all the possible business needs.

What is also important to note, and this will make the bridge into the next section, is that the returned type on this repository, as well as the type parameter for the CrudRepository is of type Product.

This Product represents the entity from our data model, as it is stored in the database. Obviously, this is defined via Java, as we will see in the next section.

Different data formats between the persistence layer and application layer

As seen earlier, the data formats between the database and application layer, can be different, and, even when they are fully identical, it is a good practice to abstract way the database representation into a format that is "coupled tighter together" with the application code.

Often, in the database, we can store additional fields that are required for the persistence to work, like primary keys, detail relationship of a certain type with others, format of specific fields in the database, etc.
However, when working at the application layer, that information is usually not the concern of the application itself, and we may not even want to handle all the database attributes in the application code.

To separate those concerns, the service actually "translates" between the different data representations, by using what is known as "Data Transfer Objects", or DTOs for short.

The main idea is that, if we fetch a Product from the database, it's a good plan to have a ProductDTO in place, somewhere at the service level, that we will use downstream.

Another important aspect of this separation of concerns at data level, is that we are in full control of the DTOs, so we can unit test our services by mocking the repository layer and asserting on the maintenance and construction of our DTOs, which makes these services very easy to test.

Let's look in depth to the differences between the database entity and the DTOs. We start with the database entity:

@Entity
public class Product implements Serializable {
    private long id;
    private String name;
    private long stock;
    private Supplier supplier;

    public Product() {
    }

    public Product(String name, long stock, Supplier supplier) {
        this.name = name;
        this.stock = stock;
        this.supplier = supplier;
    }

    @Id
    @GeneratedValue(generator = "product_id_seq", strategy = GenerationType.SEQUENCE)
    @SequenceGenerator(name = "product_id_seq", sequenceName = "product_id_seq", schema = "product", allocationSize = 1)
    @Column(name = "id", nullable = false, updatable = false)
    public long getId() {
        return id;
    }

    public void setId(long id) {
        this.id = id;
    }

    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "supplier_id", referencedColumnName = "id", nullable = false)
    public Supplier getSupplier() {
        return supplier;
    }

    public void setSupplier(Supplier supplier) {
        this.supplier = supplier;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    } 

    public long getStock() {
        return stock;
    }

    public void setStock(long stock) {
        this.stock = stock;
    }

We can see that this class, which is annotated with the @Entity annotation, really refers to a database class. We see references to ids, database table names and many other aspects that only concern the database layer. Once more, by using annotations, we can leverage SpringBoot's capabilities of performing a lot of work for us, which is what allows to use the repository pattern described earlier.

Now, obviously, when working at the application layer, we will not be concerned with any database related entities directly.

We do need to worry in the sense that the entity-annotated class will be the interface between the repository and the database, but, that's where our concerns will end. We just know, that, given a particular repository query, we will receive back in the code an instance (or a list, or set, etc...) of this entity class.

So, in order to keep the database isolated from the rest of the code and also to make our lives easier, we can create what is called a DTO class.

We can see a DTO class as the lens through which our application "sees" the database. Application code just concerns with the exact format required by the clients of our API, and one approach to capture that requirement in our code is by using a representation that matches that exactly. That is a DTO. Here's how the DTO for a product in our example domain could look like:

public class ProductDTO {

    private String name;
    private long stock;
    private String supplierName;

    public ProductDTO() {}

    public ProductDTO(String name, long stock, String supplierName) {
        this.name = name;
        this.stock = stock;
        this.supplierName = supplierName;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getSupplierName() {
        return supplierName;
    }

    public void setSupplierName(String supplierName) {
        this.supplierName = supplierName;
    }

    public long getStock() {
        return stock;
    }

    public void setStock(long stock) {
        this.stock = stock;
    }
}

This looks simpler already! And more importantly, it is also much closer to our domain and it's actually easy to work with, now that all the database baggage is out of the picture. This is crucial for flexible and maintainable code. By controlling the DTO representation entirely, we can now manage things at the application layer much easily. We can compose and extend DTOs, we can make as complex queries as we want at the database level and we know that our DTO will be able to adapt to any needs. It puts control back in our hands.

If you still recall in the beginning, our service for listing products was returning a List<ProductDTO>. This is because services care about what clients want out of the API, and repositories can worry about the database. The interface between the two worlds happens at a service level.

Let's now see exactly how to approach testing for our architecture, at a unit level and at integration level.

Considerations for testing a microservices-oriented architecture

Now that we have looked at the building blocks of our architecture in relative depth, we can address testing while following a similar approach.

Unit testing services

Let's start by looking at how we can unit test services in our setup.

We see that the service returns a DTO after manipulating a certain DB entity through a repository, so, an ideal way to unit test a service on it's own, is to mock a certain response from a repository and assert on the resulting return of the service, to ensure that its internal logic is correctly implemented. So the plan is:

We will mock the repository classes, and place them under our control by mocking expected responses;
Using this controlled data, we can also assert that our service produces the correct DTOs by asserting on its contents;
Assert that interactions with the repository class are correct, by verifying that the repository method is called only once;

Let's see how this looks in practice:

@DisplayNameGeneration(DisplayNameGenerator.ReplaceUnderscores.class)
@ExtendWith(MockitoExtension.class)
class ProductListingServiceTest {

    private ProductListingService productListingService;

    @Mock
    private ProductRepository productRepository;

    @BeforeEach
    void setUp() {
        this.productListingService = new ProductListingService(productRepository);

        Product product = new Product();
        product.setName("test");
        product.setStock(100L);
        product.setSupplier(new Supplier("someSupplier"));
        Product product2 = new Product();
        product.setName("test2");
        product.setStock(101L);
        product.setSupplier(new Supplier("someSupplier2"));
}

   @Test
    void listProduct_lists_correctly() {
        doReturn(newArrayList(product,product2).toIterable()).when(productRepository).findAll();
        List<productDTO> list = productListingService.listProducts();

        verify(productRepository, times(1)).findAll();
        assertThat(list.size(),2);
    }

We can see that by mocking the repository, we achieve the isolation level we need by testing the service internal logic exclusively.

The annotation @ExtendWith(MockitoExtension.class) is used to make sure that mocks can be configured in the test context by simply annotating them with @Mock as seen above.

Like this, we test exactly the method we need from our service and can be sure that it's internal logic is well implemented.

Let's look at integration testing with Spring's MockMvc.

Using MockMvc to write integration tests

Now, we are interested in testing our API from an endpoint perspective, i.e., from the Resource layer.

In order to do that, we can use mockMvc.

MockMvc is a Spring class we can leverage to write integration tests for our endpoints. In essence, after some minimal wiring, we can simulate a request to our API endpoints, just like it would come from an external client, and assert on its return status, value and other things to test the API in a more "e2e" fashion.

To setup MockMvc we simply need some annotations in our test class, as follows:

@AutoConfigureMockMvc
@SpringBootTest
class ProductsResourceTest {

    @Autowired
    private MockMvc mockMvc;
    (...)

We add two annotations, the @SpringBootTest that takes care of wiring for our application context, repositories and other necessary things for the application to start from within a test context, and the @AutoConfigureMockMvc that configures the MockMvc class to actually be wired correctly.

Once this is set up, we can see that the instance of mockMvc allows us to send http requests, and assert on the responses, while keeping certain external dependencies under our control.

An example of such a request could be:

@AutoConfigureMockMvc
@SpringBootTest
class ShoppingCartResourceTest {

    @Autowired
    private MockMvc mockMvc;

     @Mock
    private ProductListingService productListingService;

    @BeforeEach
    void setUp() {
        mockMvc = MockMvcBuilders.standaloneSetup(
            new ProductResource(productListingService))
            .build();
    }

    @Test
    void listProducts_with_empty_repository_returns_OK_with_empty_list() throws Exception {

        MockMvc result = mockMvc.perform(get("/list-products")
            .contentType(MediaType.APPLICATION_JSON_VALUE)
            .andExpect(status().isOk());

        assertTrue(result.andReturn().getResponse().getContentAsString().equals("{}"));
    }

Here we can exercise the entire API from a client perspective, and, we can obviously have more complex testing setups that seed test datasources with dummy data, and we can connect the Spring application to that test context data source to simulate requests with more real world data, but, this will be a subject for a future post.

Conclusion

Hopefully, after this description of a microservices architecture, now you are better equipped to see when it is a suitable solution for your project, how to strucuture an application around microservices and also had a sneak peek about different levels of testing a microservices architecture with Spring.
Suggestions/comments welcome!

Rust - understanding traits 1

Bruno Oliveira — Sat, 05 Sep 2020 20:49:21 +0000

Introduction

Many languages today are object-oriented, support this paradigm or, not being object-oriented directly, allow for the programmer to define its own data types.

Usually, we want these types to exhibit certain traits of behavior under certain circumstances, for example, when printing the types to standard output, or when making it possible to iterate over a certain type, we need to know how to perform these operations. These are defined by making sure that our types implement certain methods, usually provided by a trait.

Traits

A trait is a collection of methods defined for an unknown type: Self. They can access other methods declared in the same trait.

Traits can be implemented for any data type. In the example below, we define Animal, a group of methods:

trait Animal {
    // Static method signature; `Self` refers to the implementor type.
    fn new(name: &'static str) -> Self;

    // Instance method signatures; these will return a string.
    fn name(&self) -> &'static str;
    fn noise(&self) -> &'static str;

    // Traits can provide default method definitions.
    fn talk(&self) {
        println!("{} says {}", self.name(), self.noise());
    }
}

Having defined this trait, we can now make our own types implement this trait, which means that they will have all the behavior of an animal, while still adding their own functionality.

As examples of animals, we can define two completely distinct animal types: a StuffedAnimal and a Cat. Both are animals, so, they share certain common characteristics, like having a name, talking and making noises, but the way they do it will differ whether we are referring to a stuffed animal or to a real cat. Let's see how we can make our types implement a certain trait.

Making a custom type implement a trait

To start, we first define two very basic custom types that represent the concepts we will want to illustrate, a stuffed animal and a cat:

struct Cat {
    name: &'static str,
    age: i32
}

struct StuffedAnimal {
    name: &'static str
}

A cat here is simply represented by a name and its age, while a stuffed animal has simply a name.

For now, these types can be instantiated and used as follows:

fn main() {
  let c=Cat{name: "Bobi",age:8};
  println!("Cat's name is {} and he is {} years old",c.name, c.age);
}

So, we can leverage the attributes that represent our types to work with a higher level representation of a certain concept in code.

With traits, this idea can be expanded upon, because in essence, our types will exhibit the behavior defined by the trait that they implement. In our particular case, we will be able to "look at" our types as "animals", since they will be able to talk with a specific noise. It will also be possible to instantiate them using the new() method we see above, which will simply reference the "hand-made" struct creation we see above.

Here's how to implement a trait for a type:

impl Animal for Cat {
    // `Self` is the implementor type: `Cat`.
    fn new(name: &'static str) -> Cat {
        Cat { name: name, age: 1 }
    }

    fn name(&self) -> &'static str {
        self.name
    }

    fn noise(&self) -> &'static str {
        "Meowww"
    }

    // Default trait methods can be overridden.
    fn talk(&self) {
        // For example, we can add some quiet contemplation.
        println!("{} pauses briefly... {}", self.name, self.noise());
    }
}

The syntax to use here is of type: impl <trait name> for <type name> as the header declaration, and, like many other languages, in the definition and actual trait implementation, we define the implementation for the methods that we want to have according to the underlying implementation. As we can see, a cat meows when making noise.

Another important thing to notice here is that we can override default trait methods, in this particular example, we override the talk() method, for the specific case of a cat.

So, this is how we can define a trait for a user-defined type.

Full example

As a full example, we can also implement the trait for a stuffed animal:

trait Animal {
    // Static method signature; `Self` refers to the implementor type.
    fn new(name: &'static str) -> Self;

    // Instance method signatures; these will return a string.
    fn name(&self) -> &'static str;
    fn noise(&self) -> &'static str;

    // Traits can provide default method definitions.
    fn talk(&self) {
        println!("{} says {}", self.name(), self.noise());
    }
}

struct Cat {
    name: &'static str,
    age: i32
}

struct StuffedAnimal {
    name: &'static str
}

impl Animal for Cat {
    fn new(name: &'static str) -> Cat {
        Cat { name: name, age: 1 }
    }

    fn name(&self) -> &'static str {
        self.name
    }

    fn noise(&self) -> &'static str {
        "Meowww"
    }

    // Default trait methods can be overridden.
    fn talk(&self) {
        // For example, we can add some quiet contemplation.
        println!("{} pauses briefly... {}", self.name, self.noise());
    }
}

impl Animal for StuffedAnimal {
    fn new(name: &'static str) -> StuffedAnimal {
        StuffedAnimal { name: name}
    }

    fn name(&self) -> &'static str {
        self.name
    }

    fn noise(&self) -> &'static str {
        "<random factory noise>"
    }
}

fn main() {
  let c: Cat=Animal::new("Bobi");
  println!("Cat's name is {} {} ",c.name(), c.age);
  c.talk();

  let stuffed: StuffedAnimal=Animal::new("BobiStuffed");
  stuffed.talk();
}

So, we can see when we run this, that depending on the implementation of the trait, we can get different behavior. Note also that the talk method returns the default implementation on the second example.

Conclusion

After this basic introduction to traits, you should be able to add custom trait implementations for your own types and make your types more versatile and closer to your problem domains! Stay tuned for more Rust!

Testing web applications - adapt your tests to your architecture

Bruno Oliveira — Wed, 29 Jul 2020 14:50:37 +0000

Introduction

Testing is a key concept in modern software development. It's hard to imagine any modern, large web application of today, being developed or maintained without the support and assurance of automated tests in some way.

Automated tests are efficient because they empower developers to "move fast and break things". It's a bit like running on a suspended rope knowing that there's a net below you, that will catch you if you fall. You end up focusing more on the actual running and balance, because the fear of dire consequences if you fall is much, much lower.
With writing software the stakes are the same: you feel much more confident making a large refactor, re-working a core module in your application, making a large change in the interaction between several modules, if your tests will warn you when you break things.

Note that it's also possible to have applications and entire businesses being successful and successfully maintained without any automated tests. But, in those places, there is usually a team of QAs and additionally of "domain-expert users", testing and using your applications to ensure correctness at a technical level as well as from the "deliverables" perspective (e.g. having medical researchers/doctors using your application, if your domain is in medical sciences, telling you that "drug X can not be used when the list of medical conditions contains A, B or C, etc.)

The key takeaway will be that having automated tests in place constitutes an assurance for developers, and can actually make sure that if/when code reaches staging and/or the QA teams, that it does it with a very high certainty that the produced work is technically correct.

We need all types of tests

There are many articles online about how important it is to have multiple types of tests. This is for a good reason. Let's look at some of the characteristics of the different types of tests:

Unit tests: these are by far the most popular tests to write. They are supposed to be fast, self-contained, simple and a lot. Because they are exercising our code at its lowest level of complexity (a unit here is defined to be a single method that is part of the public interface of a class or module), these tests usually tend to be very simple and very technical. They are exercising the pieces of inner logic of a very specific class to assert the behavior of a very specific code unit. So, developers can write a lot of these tests, very fast.
Integration tests: these are a level above unit tests, and, their main purpose is to make sure that the multiple pieces of software that form our application work well together. They test the integration of our application with all the parts that live outside of our application. These are very important because they go one level above the assurance we get from unit tests: now that we know the internals work, how do the various pieces play together?
Since these tests are of a different nature, usually, they are harder to write, they run slower, and developers tend to write them a lot less, if not at all. I'd argue in the microservices paradigm most applications rely on these days, these are the most important tests. They strike the perfect balance between useful feedback and development speed/maintenance cost.
E2E tests: These are yet another kind of tests, that tend to be driven "backwards", meaning, instead of testing our application from the inside to assert it will work well from "the outside", we do the opposite here: starting from the UI (using tools like Selenium and/or human QAs interacting with the app under several scenarios), does everything work as we expect it "inside"?
These are yet another level of tests that we can use to be sure that our application works. We can argue that these are the most important/useful tests. After all, users interacting and using our application, will do it from the UI.
However, these tests are hard to write and maintain, and the feedback they offer during the development process, is very limited, and thus, these tests are often overlooked or forgotten.

It is desirable to have all kinds of tests in place if we want to ensure that our applications work correctly, but, even more importantly:

Tests are all about balance and development assurance - write the tests you need today, to both drive your application architecture and correctness to where you want it to be. You can add or change more later.

The popular test pyramid highlights the various types of tests we can rely on, together with their effectiveness as well as ROI. We see that unit tests stand at the bottom of the pyramid, they are the faster to write and run, but, work in very high isolation from the rest of the application.

This was a great view on how to write tests, but, let's not forget it first appeared in 2009 (11 years ago!!) and, over the course of the last 11 years, the way we build applications has changed a lot, and, so should our view on how and why we write tests. Let's explore some alternatives, while always keeping one thing in mind: we always need all types of tests.

Tests in modern software development

Over the course of the last 11 years, the way we view and build software has changed dramatically. So, it's only natural to think that the way we view application testing has changed with it. And it has.

As the applications evolve to become more decentralized and more distributed, the code flow within an application, depends more on the interaction between multiple services, most likely some external integrations with external APIs as well.

This means that to test a modern application, the assurance that each single, standalone service works well on its own is no longer sufficient to ensure that the application as a whole works correctly anymore.

In fact, the best way to test modern applications, relies on their architecture: a microservice-oriented architecture ends up driving the new ways that testing is done.

This type of architecture lends itself naturally to more integration tests and E2E tests instead of fully relying on simple unit tests as the main harness for correctness of inner logic. Why? Because the way this architecture works is that certain features work differently based on certain state of other (dependent) microservices.

So, it makes sense to have tests that pull in other services, test database access with certain users and/or roles, write fake data, even spin-up fully dedicated "TestApplications" whose idea is to inject mock services, or services using a test configuration, specifically used to test the application flow as a whole.

With this testing strategy, we get several benefits:

Tests that pull in other dependent services can provide much better correctness feedback and assurance, at the expense of a slightly more expensive start-up/setup time;
With Docker, it becomes possible to essentially simulate a complete environment within a certain test class or method, which can provide essential correctness feedback before moving to staging or production environments;
Tests are now closer to the real-world scenario when compared to more black-box tests, like unit ones;

With this in mind, a new trend has emerged recently (last 5 years or so) called the "inverted testing pyramid":

And, while some people argued that inverting the test pyramid is an anti-pattern (see here for a link and a useful discussion about this specific topic), I like to think that it's not an anti-pattern, it's a reflection of modern software development practices, which makes it as good as it can or needs to be for current needs of modern teams.

Frameworks like SpringBoot make writing integration tests really simple, working based on annotations and configuration files, that allows us to configure environments as close as we want to production, so, when writing endpoints, the base go-to test should be a simple integration test that tests the base logic of the endpoint together with authentication, response expectations, request bodies, etc, using other services as needed (for example, a UserService to inject fake credentials in a request requiring a special authentication, etc.) and, only if the service itself contains some very complex inner logic, then we must add unit tests to cover it as well.

The idea for modern software testing seems to be:

Write fewer, but more representative tests, i.e. some unit, not a lot, mostly integration, some E2E

However, as always, it all depends on your current codebase, technologies being used, amount of legacy code that is untested, etc.

Use the best level of testing that is right for your application, and, focus on writing only the most important tests.

Kent C. Dodds has popularized what is now known as the test trophy, applied to JS:

Obviously, using a statically typed language can already provide us with the base of the trophy "for free", but the basic concepts still apply: we need to write less, but, more meaningful and useful tests.

Integration tests are essential to ensure units collaborate correctly, and, in this case, a unit can be a specific endpoint that requires the work of multiple services to exercise its purpose.

Unit tests are at the lower-level of abstraction from the code itself, and, they are small, self-contained and fast, and we can use them sparingly to ensure the correct implementation of business logic or some complex inner logic "hidden" inside a certain service.

E2E can be seen as less valuable, especially when we can have QA teams or people who are domain-experts testing our applications, much like real users would, but, it is very important to do them, whether automated or not.

Conclusion

Modern software development has changed the role and importance of testing, and while we looked at several types of testing, and we saw how cost-effective vs. benefit they can be, we saw that there has been a paradigm shift on how modern companies approach testing.

With the tools available at our disposal, we can rely on more representative and meaningful tests to ensure we deliver the maximum possible value.

Writing a mathematical expression evaluator in Java

Bruno Oliveira — Fri, 26 Jun 2020 08:54:14 +0000

Introduction

Writing an expression evaluator, as simple as it may be, has always been a long standing challenge for me, mostly because it feels so simple, and yet, gets tricky very fast.

Writing code to parse and evaluate a simple expression with few operands and/or operators with the same precedence, is simple. However, the generalization to higher complexities can be really tricky due to multiple factors: operator precedence, expression nesting and parenthesis.

Inspired by the first chapter of what promises to be an amazing read of the book: "Engineering a Compiler", 2nd edition, and, taking only their simple definitions of a parser and a scanner, I managed to put together a small example that seems good enough to share.

Devising a language to talk about mathematical expressions

The first thing we need to define when writing something like this, is a language, that translates well into code, that allows us to think about the problem at a higher level of abstraction.

We will say that an expression is composed of tokens, or, as I called it, TokenTypes.

public enum TokenType {
  ADD,
  SUB,
  MUL,
  DIV,
  POW,
  LPAR,
  RPAR,
  VALUE;

  @Override
        public String toString() {
            switch (this.ordinal()){
            case 0:
                return "+";
            case 1:
                return "-";
            case 2:
              return "*";
            case 3:
              return "/";
            case 4:
              return "^";
            case 5:
              return "(";
            case 6:
              return ")";
            case 7:
              return this.name();
            default:
              return "null";
            }
        }

  public static TokenType fromString(String s){
    switch (s){
      case "+":
      return TokenType.ADD;
      case "-":
      return TokenType.SUB;
      case "*":
      return TokenType.MUL;
      case "/":
      return TokenType.DIV;
      case "^":
      return TokenType.POW;
      case "(":
      return TokenType.LPAR;
      case ")":
      return TokenType.RPAR;
      default:
      return TokenType.VALUE;
    }
  }
}

This is a simple Java enum with methods to write from/to Strings into Tokens and vice-versa.

We immediately notice that the . for decimal values is encoded in the enum as being a "value" since it's not a special token. Obviously this can be improved further, but, seems good enough for now.

So, from now on, when we talk about a mathematical expression, we will talk about it in terms of its TokenTypes.

Writing and understanding the Scanner

The scanner is the piece in a compiler front-end that will read an expression and match its pieces to known tokens, so that the expression can have a real meaning in the context where we will be evaluating it.

Starting with a simple expression like 12+5, after a Scanner pass, we could get output along the lines of:

(12, Token.NUM) , (+, Token.ADD), (5, Token.NUM)

This will enable us to look at the expression and parse it, evaluate it, reduce it to a simpler form, etc.

In the first Scanner pass, some things will need to be corrected, for instance, if we see the expression:

(-, Token.SUB), (3, Token.NUM), what we really want to have is: (-3, Token.NUM).

I chose to do this further down the pipeline, in the Parser stage.

The code for the Scanner is as follows:

public class ScannedToken {
  private String expressionPiece;
  private TokenType type;

  public ScannedToken(String exp, TokenType type){
    this.expressionPiece = exp;
    this.type = type;
  }

  @Override
  public String toString(){
    return "(Expr:"+ expressionPiece+ ", Token:"+ type+")";
  }

  public TokenType type(){
    return type;
  }

  public String expression(){
    return expressionPiece;
  }

}

A ScannedToken is a token annotated with the tokenType, which we will use when performing an actual expression scan.

The code for the Scanner is as follows:

 import java.text.DecimalFormat;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;

public class Scanner {

    private final String expression;

    public Scanner(String expr) {
        this.expression = expr;
    }

    public List<ScannedToken> scan() {
        StringBuilder value = new StringBuilder();
        List<ScannedToken> scannedExpr = new ArrayList<>();
        for (char c : expression.toCharArray()) {
            TokenType type = TokenType.fromString(new String(new char[] {c}));
            if (!type.equals(TokenType.VALUE)) {
                if (value.length() > 0) {
                    //Add the full value TOKEN
                    ScannedToken st = new ScannedToken(value.toString(), TokenType.VALUE);
                    scannedExpr.add(st);
                }
                value = new StringBuilder(new String(new char[] {c}));
                ScannedToken st = new ScannedToken(value.toString(), type);
                scannedExpr.add(st);
                value = new StringBuilder();
            } else {
                value.append(new String(new char[] {c}));

            }
        }
        if (value.length() > 0) {
            //Add the full value TOKEN
            ScannedToken st = new ScannedToken(value.toString(), TokenType.VALUE);
            scannedExpr.add(st);
        }

        return scannedExpr;
    }

    public double evaluate(List<ScannedToken> tokenizedExpression) {

        if (tokenizedExpression.size() == 1) {
            return Double.parseDouble(tokenizedExpression.get(0).expression());
        }
        //Eval order is PEMDAS - Parenthesis, exponents, multiply, divide, add, subtract
        List<ScannedToken> simpleExpr = new ArrayList<>();

        int idx =
            tokenizedExpression.stream()
                .map(ScannedToken::type)
                .collect(Collectors.toList())
                .lastIndexOf(TokenType.LPAR);
        int matchingRPAR = -1;
        if (idx >= 0) {
            for (int i = idx + 1; i < tokenizedExpression.size(); i++) {
                ScannedToken curr = tokenizedExpression.get(i);
                if (curr.type() == TokenType.RPAR) {
                    matchingRPAR = i;
                    break;
                } else {
                    simpleExpr.add(tokenizedExpression.get(i));
                }
            }
        } else {
            simpleExpr.addAll(tokenizedExpression);
            return evaluateSimpleExpression(tokenizedExpression);
        }

        double value = evaluateSimpleExpression(simpleExpr);
     //   System.out.println("val is " + value);
        List<ScannedToken> partiallyEvaluatedExpression = new ArrayList<>();
        for (int i = 0; i < idx; i++) {
            partiallyEvaluatedExpression.add(tokenizedExpression.get(i));
        }
        partiallyEvaluatedExpression.add(new ScannedToken(Double.toString(value), TokenType.VALUE));
        for (int i = matchingRPAR + 1; i < tokenizedExpression.size(); i++) {
            partiallyEvaluatedExpression.add(tokenizedExpression.get(i));
        }

        // from idx find first ), extract, evaluate, replace, call recursively
      //  System.out.println("Expr to eval indexes: " + idx + ", " + matchingRPAR);
        System.out.println(partiallyEvaluatedExpression);
        return evaluate(partiallyEvaluatedExpression);
    }

    //A simple expression won't contain parenthesis
    public double evaluateSimpleExpression(List<ScannedToken> expression) {
        if (expression.size() == 1) {
            return Double.parseDouble(expression.get(0).expression());
        } else {
            List<ScannedToken> newExpression = new ArrayList<>();
            int idx = expression.stream().map(ScannedToken::type).collect(Collectors.toList()).indexOf(TokenType.POW);
            if (idx != -1) {
                double base = Double.parseDouble(expression.get(idx - 1).expression());
                double exp = Double.parseDouble(expression.get(idx + 1).expression());
                DecimalFormat df = new DecimalFormat(".00");
                double ans = Math.pow(base, exp);
                for (int i = 0; i < idx - 1; i++) {
                    newExpression.add(expression.get(i));
                }
                newExpression.add(new ScannedToken(ans + "", TokenType.VALUE));
                for (int i = idx + 2; i < expression.size(); i++) {
                    newExpression.add(expression.get(i));
                }
                return evaluateSimpleExpression(newExpression);
            } else {
                int mulIdx = expression.stream()
                    .map(ScannedToken::type)
                    .collect(Collectors.toList())
                    .indexOf(TokenType.MUL);
                int divIdx = expression.stream()
                    .map(ScannedToken::type)
                    .collect(Collectors.toList())
                    .indexOf(TokenType.DIV);
                int computationIdx = (mulIdx >= 0 && divIdx >= 0) ? Math.min(mulIdx, divIdx) : Math.max(mulIdx, divIdx);
                if (computationIdx != -1) {
                    double left = Double.parseDouble(expression.get(computationIdx - 1).expression());
                    double right = Double.parseDouble(expression.get(computationIdx + 1).expression());
                    DecimalFormat df = new DecimalFormat(".00");
                    double ans = computationIdx == mulIdx ? left * right : left / right * 1.0;
                    for (int i = 0; i < computationIdx - 1; i++) {
                        newExpression.add(expression.get(i));
                    }
                    newExpression.add(new ScannedToken(ans + "", TokenType.VALUE));
                    for (int i = computationIdx + 2; i < expression.size(); i++) {
                        newExpression.add(expression.get(i));
                    }
                    return evaluateSimpleExpression(newExpression);
                } else {
                    int addIdx = expression.stream()
                        .map(e -> e.type())
                        .collect(Collectors.toList())
                        .indexOf(TokenType.ADD);
                    int subIdx = expression.stream()
                        .map(e -> e.type())
                        .collect(Collectors.toList())
                        .indexOf(TokenType.SUB);
                    int computationIdx2 = (addIdx >= 0 && subIdx >= 0) ?
                        Math.min(addIdx, subIdx) :
                        Math.max(addIdx, subIdx);
                    if (computationIdx2 != -1) {
                        double left = Double.parseDouble(expression.get(computationIdx2 - 1).expression());
                        double right = Double.parseDouble(expression.get(computationIdx2 + 1).expression());
                        DecimalFormat df = new DecimalFormat(".00");
                        double ans = computationIdx2 == addIdx ? left + right : (left - right) * 1.0;
                        for (int i = 0; i < computationIdx2 - 1; i++) {
                            newExpression.add(expression.get(i));
                        }
                        newExpression.add(new ScannedToken(ans + "", TokenType.VALUE));
                        for (int i = computationIdx2 + 2; i < expression.size(); i++) {
                            newExpression.add(expression.get(i));
                        }
                        return evaluateSimpleExpression(newExpression);
                    }
                }

            }
        }
        return -1.0;
    }
}

The code for the scanner is a bit too convoluted, but, the basic idea is to look at an expression and evaluate it like a human mathematician would:

We look for the innermost parenthesis, extract its inner expression and consider it a Simple expression. So a simple expression is an expression without parenthesis.

A simple expression can be evaluated directly, via recursion, following the PEMDAS rule: Parenthesis first, then exponents, then multiplication and division in the order which they appear and then subtraction and additions in the order they appear.

Once simple expressions are evaluated, the resulting token is replaced in the original expression, and recursively passed to be evaluated again, until the end result is a single token with a single value.

And this is the basic idea.

Adding a Parser to avoid issues with "incomplete" tokens

Obviously, the way this is implemented now, it's still incomplete, both in terms of error handling and wrongly tokenized expressions. Error handling hasn't been done, so things like incomplete expressions or unbalanced parenthesis will either take the process into a loop or just crash. To do later :)

What we need to do now in the parser, to get a simple solution working, is to decide when can we create a negative number token from a pair of Token.SUB and Token.VALUE.

The edge cases here are two:

if Token.SUB and Token.VALUE are in the beginning of the expression, then we create a single token with a negative value;
if these two tokens are right after an opening parenthesis, they are part of the beginning of a complex expression on which case we need to unify them as well;

This is what we do in the parser:

import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
public class Parser {

    private final List<ScannedToken> expression;

    public Parser(List<ScannedToken> expression) {
        this.expression = expression;
    }

    /*
    We need a triple in a sliding window fashion where middle is current token so we can contextualize what
    needs to be parsed into correct tokens
     */
    public List<ScannedToken> parse() {
        TokenType prev = null;
        TokenType curr = null;
        TokenType next = null;

        List<ScannedToken> properlyParsedExpression = new ArrayList<>();

        List<TokenType> types = expression.stream().map(ScannedToken::type).collect(Collectors.toList());
        List<Integer> indexes = new ArrayList<>();
        List<ScannedToken> negativeValues = new ArrayList<>();

        for (int i = 0; i < types.size() - 1; i++) {
            prev = i == 0 ? null : types.get(i - 1);
            curr = types.get(i);
            next = types.get(i + 1);
            if (prev == null && curr == TokenType.SUB && next == TokenType.VALUE) {
                ScannedToken negativeValue =
                    new ScannedToken("" + (-1 * Double.parseDouble(expression.get(i + 1).expression())),
                        TokenType.VALUE);
                System.out.println("new token at index " + i);
                indexes.add(i);
                negativeValues.add(negativeValue);

            } else if (prev == TokenType.LPAR && curr == TokenType.SUB && next == TokenType.VALUE) {
                ScannedToken negativeValue =
                    new ScannedToken("" + (-1 * Double.parseDouble(expression.get(i + 1).expression())),
                        TokenType.VALUE);
                System.out.println("new token at index " + i);
                indexes.add(i);
                negativeValues.add(negativeValue);
            }
        }

        int maxIterations = expression.size();
        int i = 0;
        int j = 0;
        while (i < maxIterations) {
            if (indexes.contains(i) && j < negativeValues.size()) {
                properlyParsedExpression.add(negativeValues.get(j));
                j++;
                i++;
            }
            else {
                properlyParsedExpression.add(expression.get(i));
            }
            i++;
        }
        System.out.println(properlyParsedExpression);
        return properlyParsedExpression;
    }

}

And, finally the Main function of our code, where we execute what we did:

import java.util.List;
import java.io.BufferedReader; 
import java.io.IOException; 
import java.io.InputStreamReader;

class Main {
  public static void main(String[] args) throws IOException{
    System.out.println("Enter a mathematical expression");
    //Enter data using BufferReader 
    while(true){
        BufferedReader reader =  
                   new BufferedReader(new InputStreamReader(System.in)); 

        // Reading data using readLine 
        String name = reader.readLine(); 

    Scanner sc = new Scanner(name);
    //(12*5)+1*(8.6-3*2^2)-34
    //(12*5)
    //12*5
    //12
    //-5+12*(-3+2)
    //-12
    //12
    //15/(3+2)

    List<ScannedToken> scanExp = sc.scan();
    Parser parser = new Parser(scanExp);
    List<ScannedToken> parsed = parser.parse();
    scanExp.forEach(e->System.out.println(e));
    System.out.println(sc.evaluate(parsed));
    }
  }
}

So, we evaluate the parsed expression, with the corrected tokens, and we get the value we want.

Additional design considerations

There are many additional design considerations to take that can improve this:

Error handling is inexistent, so, if there is an invalid expression, the scanner will just die with a stackoverflow or outOfBounds exception;
The parser needs to be more robust to handle wrong inputs graciously;
Obviously, this is an approach that mimics how a human would evaluate an expression, and the grammar rules, which are in essence the precedence rules of operations are all encoded in the Scanner code by following the PEMDAS rule programatically; In a real implementation, we can assign for example precedence values to the operators and build a tree that takes these precedences into account, maybe delegating this step to the parser, and then evaluating the expression is as easy as doing an in-order tree traversal by priority values recursively evaluating sub-expressions;

Conclusion

This was a great exercise and I think it's a nice challenge that everyone can take on! It shows how something which is deceptively simple can actually be quite tricky and it's a humbling exercise!
As a recommended read from where I got the inspiration to write this, I recommend the 2nd edition of the book "Engineering a Compiler"

DEV Community: Bruno Oliveira

Basic RAG app with Spring AI, Docker and Ollama

Technology Stack

The app

How does it really work?

Diving into the code

First, there was a RAG database

Then, PGVectorStore enters the chat

Docker-compose based Springboot configuration

Ollama docker container to pull the models so they are locally available

Java code to tie it all together

Conclusion

Announcement: Book release!!

Parsing and reading Excel binary data using Pandas

Go broader to go further - leverage your learning chances

Building Kubernetes-first apps: yay or nay?

Practical Java 16 - Using Jackson to serialize Records

A reflection on modern software development

Introduction

Accidental and inherent complexity

Taming complexity: a toolset to be efficient in the modern software development age

Teams and processes

Ride the devOps wave

Writing good code

Conclusion

Testing Amazon S3 using s3Ninja

Architecturing Spring services

Rust - understanding traits 1

Testing web applications - adapt your tests to your architecture

Writing a mathematical expression evaluator in Java

Then, `PGVectorStore` enters the chat