DEV Community: Andrew Panfilov

Chatbot Prototype: Architectural Proposal

Andrew Panfilov — Wed, 22 May 2024 11:46:23 +0000

Intro

This article presents an example of an architectural proposal for creating an intelligent chatbot prototype with LLM. It can serve as useful learning material for those who have never written such documents, and for those who have written similar proposals, it may enrich their experience. In any case, I would be glad to hear readers' comments and discuss what should be added to or corrected in this document.

Task Definition

For a large university, it is necessary to create a chatbot to which employees can ask questions in free form regarding students, attendance, statistics, aggregation, and various database queries about students. The university attempted to create a system based on Text-to-SQL on its own. However, in three out of ten cases, the generated SQL query returned incorrect results, and in seven out of ten cases, it returned correct results. This accuracy level does not satisfy the university, so a prototype chatbot needs to be created to respond to university employees’ questions about students with 100% accuracy. This proposal describes a system capable of providing the most accurate information about students from the university database.

Analysis

Terms and Acronyms

Component refers to a runtime entity, it can be deployed independently.
Deployable artifact is the application code as it runs on production: compiled, built, bundled, minified, optimized, and so on. Most often, it's a single binary or a bunch of files compressed in an archive. One can store and version an artifact. An artifact should be configurable in order to be deployable in any environment. For example, if one needs to deploy to staging and production servers, one should be able to use the same artifact. Only the configuration must change, not the artifact itself.
Personal Identifiable Information (PII) refers to any data that could potentially identify a specific individual. This information can include direct identifiers, such as name, social security number, driver’s license number, and passport number, which can directly recognize an individual. It also encompasses indirect identifiers when combined with other information, such as date of birth, place of birth, and mother’s maiden name, that can be used to deduce an individual's identity.
Docker is a set of a platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers.
Container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.
Docker Compose is a tool designed for defining and orchestrating multi-container Docker applications. It allows users to use a YAML file to configure an application’s services, networks, and volumes. By utilizing a single command, users can then initiate and activate all the services outlined in their configuration. It streamlines the deployment process, enabling the management of the entire lifecycle of an application through straightforward commands.
SQL query is a request for data or information from a database table or combination of tables in SQL (Structured Query Language). This language is used for managing and manipulating relational databases. SQL queries can perform a variety of tasks, including retrieving specific information from a database, inserting new data, updating existing data, and deleting data. Queries are constructed using specific SQL syntax and can range from simple commands to retrieve all records from a database table to complex queries involving multiple operations like joins, subqueries, and aggregate functions to extract and manipulate data according to specific requirements.
Chatbot is a software application designed to simulate conversation with human users, especially over the Internet. Chatbots can function in a wide range of environments, including websites, mobile apps, messaging platforms, and telephone services. They are used for various purposes such as customer service, information acquisition, and entertainment. Chatbots can be simple, based on pre-defined scripts to handle specific tasks, or more complex, utilizing natural language processing and artificial intelligence to engage in more open-ended conversations and learn from interactions to improve their responses over time.
Snowflake database refers to Snowflake Inc.'s cloud-based data warehousing service. It is a fully managed service that allows organizations to store and analyze data using cloud-based hardware and software. Snowflake's architecture is unique because it separates storage and compute, enabling users to scale storage and compute independently. This means organizations can adjust their storage capacity and computational power based on their current needs without affecting the other.
SPA (Single Page Application) is a web application or website that interacts with the user by dynamically rewriting the current page rather than loading entire new pages from the server. This approach enables a more fluid and faster user experience, as it minimizes the amount of data transferred between the server and the client, reduces loading times, and provides a seamless interaction similar to desktop applications. SPAs use AJAX and HTML5 to asynchronously load content and update the webpage with new data from the web server, without the need for page refreshes.

Requirements

This article uses system-centric requirements, which are not the same as user-centric user stories and may not contain any business value. At the same time, it is not a contradiction or opposite approach to user-centric user stories – a user story that mentions business value may contain several system-centric requirements.

Additionally, it is essential to understand the difference between functional and non-functional requirements.

From https://en.wikipedia.org/wiki/Non-functional_requirement:

“In systems engineering and requirements engineering, a non-functional requirement (NFR) is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. They are contrasted with functional requirements that define specific behavior or functions. The plan for implementing functional requirements is detailed in the system design. The plan for implementing non-functional requirements is detailed in the system architecture, because they are usually architecturally significant requirements.

Broadly, functional requirements define what a system is supposed to do and non-functional requirements define how a system is supposed to be. Functional requirements are usually in the form of "system shall do ", an individual action or part of the system, perhaps explicitly in the sense of a mathematical function, a black box description input, output, process and control functional model or IPO Model. In contrast, non-functional requirements are in the form of "system shall be ", an overall property of the system as a whole or of a particular aspect and not a specific function. The system's overall properties commonly mark the difference between whether the development project has succeeded or failed.

Non-functional requirements are often called "quality attributes" of a system. Other terms for non-functional requirements are "qualities", "quality goals", "quality of service requirements", "constraints", "non-behavioral requirements", or "technical requirements". Informally these are sometimes called the "ilities", from attributes like stability and portability. Qualities—that is no n-functional requirements—can be divided into two main categories:

1) Execution qualities, such as safety, security and usability, which are observable during operation (at run time).

2) Evolution qualities, such as testability, maintainability, extensibility and scalability, which are embodied in the static structure of the system.”

Principles:

Written once, read many times
1. Specifications of an IT system (like other texts), are written once and are read many times over the lifespan of the system. Hence two observations become clear: reading time matters more than writing time, and upfront quality is free. I.e. it makes sense to optimize the text for readability and comprehensibility, even if that means the author has to put in more effort.
No Nobel prize in literature
1. Specifications are technical texts written for one purpose only: the successful transfer of meaning from one person to another, over distance and time. The author will never win a Nobel prize in literature for it. Therefore it does not make sense to put the effort into an aesthetic, pleasurable text.
Specifications are for systems
1. System specifications describe what is required from the system. This should not at all prevent you from describing what the users do (in the sense of a business process description), but you want to do that in other parts of the documentation (use cases, for instance).
Detailed written documentation has very low communication effectiveness
1. Therefore writing detailed functional requirements might not be the best of ideas. Consider NOT doing it if you expect a high rate of requirements creeps. Assume a change rate of 2-3% per month if you do not know the specific change rate of your project (which actually IS high, though widely experienced).

Here is a well-structured explanation of a requirements format:

Large image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6pz78jmxtw2uki01afre.png

Links:

1. User's authentication

The system must provide a user with the capability to request a student related information in free-form textual communication with the chatbot if the following condition is true:

the user was successfully logged in with the provided login password

2. Free-form request's bucketing

The system must recognize to what predefined SQL query relates a free-form textual request about student information if all of the following conditions are true:

The user sent the request through the chatbot
The request semantically belongs to student information

3. Recognition failure

The system must reply to a user with recognition failure if all of the following conditions are true:

The user sent the request through the chatbot
The request semantically doesn't belong to any predefined SQL queries that existed in the system

4. Correctness confirmation recognition

The system must provide a user with the capability to confirm the correctness of predefined SQL query recognition if all of the following conditions are true:

The user sent the request through the chatbot
The request semantically belongs to a predefined SQL query that existed in the system
The chatbot sent a confirmation question to the user, and the user confirmed that the free-form request belongs to the recognized SQL query

5. Additional parameters input

The system must provide a user with the capability to input additional parameters for the recognized SQL query in a dialogue with the chatbot if the following condition is true:

Chatbot successfully recognized predefined SQL query based on a free-form textual request from the user

6. Predefined SQL query results

The system must reply to a user with a result of a predefined SQL query invocation to Snowflake in tabular textual format in a chatbot dialogue if all of the following conditions are true:

The user sent the request through the chatbot
The request was recognized as belonging to one of the predefined SQL queries
The predefined SQL query was sent to the Snowflake database
Snowflake database responded with non-empty data

Predefined queries to Snowflake database

Student missed 3 or more periods in prior instructional week
Student misses same day of the week multiple times
Student Suspended

Design

How to read a component diagram

For cognitive load reduction, a minimalist visual language is used to depict a static structure of deployable artifacts (called components) with links between them.

Here is a subset of visual means of the standard UML component diagram, with only four elements: component, interface, and two types of connections: one for "provides" semantics and one for "uses" semantics.

This type of diagram does not allow any visual means to be treated as flows. No arrows are permitted. A sequence diagram should depict time-related communications, protocols, and data flow.

Chatbot AI Prototype components diagram

Large image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9mprx3z9jm82rkihe9wc.png

To simplify the prototype creation and ensure an efficient setup process, all the back-end components of the system will be deployed on a single machine using Docker Compose.

Components:

User's Browser: The entry point for the user's interaction with the ChatBot SPA. It's where the user types their queries in free-form text.
Nginx: Acts as a reverse proxy, load balancer, and static asset server. It also handles basic authentication using login/password to ensure only authorized users can interact with the ChatBot.
ChatBot API: The core component that processes user requests. It's a stateless backend service, possibly written in Python or NodeJS, that communicates with both the OpenAI API for natural language understanding and the Snowflake database for data retrieval. It also interacts with the Postgres database to fetch SQL query templates.
Postgres: Stores SQL query templates with placeholders. These templates are used by the ChatBot to construct SQL queries based on the user's request and additional parameters provided during the dialogue.
OpenAI API: Provides the natural language processing capabilities required to understand the user's free-form text requests and map them to the appropriate SQL query template from the Postgres database.
Snowflake: The primary data storage where actual student attendance data is kept. Once the ChatBot constructs an SQL query, it's executed against this database to retrieve the requested information.

Chatbot AI Prototype process flow

Large image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/chxjvl5gegv4kbo4kwir.png

The sequence diagram outlines the process flow of a user interacting with a ChatBot to retrieve data from a Snowflake database, using predefined SQL query templates stored in a Postgres database. The interaction involves several phases, from the initial request to the display of results.

This sequence diagram does not contain a process of preparation and filling Postgres with predefined queries with templates with descriptions and embeddings.

Here's a breakdown of each phase:

1. Initial Phase

The user inputs the chatbot's web address into their browser. The browser requests the Single Page Application (SPA) for the chatbot from the Nginx server. Nginx returns the necessary SPA assets (HTML, JavaScript, CSS) to the browser. The user types a free-form textual message into the chatbot interface. The browser sends this message to the chatbot through Nginx, which forwards the authenticated request with the message to the ChatBot API.

2. Predefined Query Recognition Phase

The ChatBot API requests an embedding vector for the message from the OpenAI API. The ChatBot API component asks Postgres for the closest predefined query description based on the user's message. After receiving the query description, the ChatBot API requests a dialogue descriptor from Postgres. The API constructs a prompt using the dialogue descriptor, predefined query description, and user's message. The prompt is sent to the OpenAI API to generate a next dialogue replica, which is used to confirm the correctness of the query recognition with the user.

3. Predefined Query Parameters Fill Phase

Upon user confirmation, the ChatBot API fetches the SQL template for the recognized predefined query from Postgres. The API requests a specific dialogue descriptor related to the predefined query. Using this descriptor, the API builds another prompt and sends it to the OpenAI API to generate dialogue for requesting additional parameters from the user. The user supplies the additional parameters requested by the chatbot.

4. Predefined Query Invocation Phase

The ChatBot API component constructs the final SQL query for invocation in Snowflake using the template filled with parameters provided by the user. This SQL query is executed against the Snowflake database. Snowflake returns the query results, which the ChatBot API formats and sends back to the browser to be displayed to the user.

Tasks breakdown

Predefined Snowflake SQL queries preparation
1. Query templates composing.
2. Embeddings generation.
Predefined Snowflake SQL descriptions preparation.
Postgres DDL composing (example)
1. vector extension setup.
2. DDL for documents table.
3. DDL for document_sections table.
4. DDL for chatbot_dialogues table.
5. DDL for llm_conversations table.
Docker-compose.yml composing.
ChatBot API back-end
1. GitHub repository with CI/CD (GitHub Actions) preparation.
2. Initial codebase composing.
ChatBot SPA
1. GitHub repository with CI/CD (GitHub Actions) preparation.
2. Initial codebase composing.
Nginx configuration composing.
Provisioning
1. GitHub repository with CI/CD (GitHub Actions) preparation.
2. Terraform scripts composing.
3. Hetzner VPS setup.

💡 How to fail as a CTO: developers' fungibility

Andrew Panfilov — Wed, 22 May 2024 11:07:00 +0000

When individual developers can be substituted for one another without impacting productivity or project outcomes:

✅ You have a homogeneous team culture; everyone works in the office and no hybrid/remote work
✅ Your dev team uses mature mainstream technologies (top TIOBE languages like Python/TypeScript/Java, relational databases like Postgres for persistence, etc.)
✅ Your teams' domains are pretty similar: developers working on one domain understand terms and acronyms from another domain
✅ You have a well-structured hiring process, and you are sure of the cultural fit
✅ You have a low churn rate: institutional knowledge is high within the engineering organization
✅ All of your engineers are experienced ones: seniors and higher
✅ You have a decent test coverage for your codebase
✅ You are sure that you have the ubiquitous language in the organization: your QA folks understand business folks and vice versa if needed

When can not be interchanged:

⛔️ You are a highly dynamic startup before product market fit
⛔️ Your organization is growing in headcount
⛔️ You have more than two programming languages for the backend, more than two for frontend, you use a young (less than 16 years old) non-SQL database or several different ones
⛔️ Your engineers follow different schools of thought: some people like objected-oriented, some functional programming
⛔️ You have vague test coverage and are not sure about the quality gates
⛔️ You have a geographically distributed team, hybrid or remote mode
⛔️ You have developers in the team who have no experience with your current programming language nevertheless of seniority
⛔️ You have two types of developers: core and provided by outsourcing or outstaffing service
⛔️ Your business works with a domain with high essential complexity

Similarities of a startup and museum of art. The role of an architect.

Andrew Panfilov — Fri, 03 May 2024 12:17:35 +0000

This post was originally published here: https://www.linkedin.com/pulse/similarities-startup-museum-art-role-architect-andrew-panfilov-fhv0f/

In a museum of the art context, comparing technical roles to positions within the museum can help illustrate each role's responsibilities and impact within their respective environments.

🖼️ Museum Director -> 💻 Chief Technology Officer (CTO):

The museum director oversees all operations, strategy, and long-term vision, ensuring it fulfills its mission, engages the public, and operates efficiently. Similarly, the CTO in a tech company is responsible for the company's technological direction and innovation and ensuring the technology strategy aligns with the company's goals. They oversee the development and dissemination of technology to enhance product offerings and improve operational efficiency, much like the museum director ensures the museum's offerings (exhibitions, collections) are relevant, engaging, and accessible.

🖼️ Exhibition Curator -> 💻 Software Architect:

An exhibition curator is responsible for selecting and organizing artworks to create exhibitions that engage visitors, tell a story, or explore specific themes or historical periods. This role involves research, understanding the audience, and designing the layout of exhibitions to guide visitors' experiences. The software architect, similarly, designs the structure of software systems, choosing the right technologies and patterns to ensure the software meets requirements and is scalable, maintainable, and efficient. Both roles require a deep understanding of their field, creativity in presentation or design, and the ability to envision and execute a coherent and engaging experience.

🖼️ Artist -> 💻 Development Team:

Artists create artworks that are the core of any museum's mission, using their skills, creativity, and vision to produce pieces that engage, challenge, and delight audiences. In the tech world, the development team, comprising software developers, engineers, and programmers, is responsible for writing the code that makes software products and services functional. They bring the vision of the product owners, CTOs, and software architects to life through technical implementation, much like artists realize their creative visions through their chosen mediums. Both contribute the essential creative and technical skills necessary for the success of their respective institutions or projects.

These analogies help understand each member's pivotal roles in their respective domains, highlighting the importance of leadership, vision, creativity, and execution in achieving successful outcomes.

💡How to fail as a CTO: never check your hiring managers

Andrew Panfilov — Fri, 03 May 2024 11:34:05 +0000

This post was originally published here https://www.linkedin.com/pulse/how-fail-cto-never-check-your-hiring-managers-andrew-panfilov-slhff/

To be a successful CTO, it is crucial to prioritize hiring and overseeing the hiring process. Neglecting this responsibility can pose a significant risk to the company.

As a confident solution, I recommend that CTOs take a more active role in the hiring process to ensure that candidates are evaluated based on their skills and experience rather than their ability to memorize a book. Today's hiring process prioritizes form over substance due to a lack of attention from CTOs towards the hiring process. Candidates for an engineering manager position are often required to memorise a book such as 'The Software Engineering Manager Interview Guide', which provides hiring managers with expected answers. The organization has appointed an engineering manager who possesses the correct answers knowledge but lacks the other two crucial components of the triad: knowledge-ability-skill.

The issue of hiring for technical positions is plagued by a common problem. A system design interview is a standard part of the hiring process for any technical position in today's job market. However, it can be problematic if the hiring manager expects a response that aligns with the book let’s say with title 'System Design Interview'. A candidate's response to a question such as 'design a payment system' that differs from what is written in the book does not necessarily make the engineer unsuitable. Hiring managers should consider the candidate's reasoning and approach before making a decision. It is important to note that not all payment systems in the world are designed as described in that book, and there are many valid reasons for this. Hiring someone who has memorized answers from books or blog posts but lacks an engineering perspective and the skills to build systems poses a significant risk.

Let's consider an example that commonly occurs during the hiring process. When a candidate is interviewing for an architect position, they may be asked questions as if they were interviewing for a delivery manager position. Similarly, if a candidate is interviewing for an engineering manager position, they may be asked questions as if they were interviewing for a staff engineer or architect position. This gives the impression that companies do not fully understand the differences between these roles. When people within these companies, such as hiring managers, create job descriptions, they may not fully comprehend the specific requirements of each role, such as the differences between a software architect and an engineering manager. Instead, they may look at job descriptions from competitors and attempt to create something similar for a position in their own company without accurately describing the unique problems that the company is facing and the responsibilities that the hired candidate will need to undertake.

An architect's primary role is technical. They create diagrams representing the system's structure, including the services' communication, order, and purpose. On the other hand, an engineering manager's position involves people management, such as hiring, performance reviews, conflict resolution, and promoting work-life balance.

It is important to note that while an architect can mentor development teams, it is not the same as people management. Having an architect present during sprint planning and demos is crucial to prevent duplication of work between teams. The architect acts as a facilitator between development teams, business representatives, management, and developers and communicates with both parties. Moreover, the architect should work closely with the product manager to participate in quarterly planning and communicate the team's current technical debt situation. The architect will discuss the team's capacity for addressing technical debt with the product manager.

The architect establishes a process for making architectural decisions within and across the organization. The development team receives a set of non-functional requirements from the architect. The development team submits architectural proposal documents to the architect or architecture team. The team outlines their vision for the current and proposed system changes in these documents. Additionally, they explain the motivation for adding a new Deployment Artifact or service and how it will communicate with the existing system, such as through REST API, PubSub Messaging, a shared file system, or a shared database. The architect must communicate the Non-Functional Requirements (NFRs) to the product manager and the product vertical within the organization. It is crucial to explain why these requirements are essential and the potential business risks that may arise if they are unmet.

The role of an engineering manager is to focus on people management. This involves having one-on-one meetings with each team member to give and receive feedback. The aim is to ensure everyone has a comfortable and efficient work environment. This helps identify and address issues early on before they become significant problems. The engineering manager should also maintain a good delivery pace while ensuring that employees are productive without burning out. Why is this important? If employees are hired correctly and happy in their roles, they bring business value and reduce the churn rate. This means that employees resign less frequently, institutional knowledge within teams grows, and programmers better understand the business beyond just the technological aspect of the product. This diversity of knowledge and understanding can provide unique insights into problems and potential solutions that product managers may not immediately realize.

The hiring process varies depending on the organization. Some companies prioritize avoiding hiring the wrong person, considering it more crucial than hiring the right person. Others prioritize hiring the right person, even taking on the risk of hiring someone who might not work out. Regardless, the CTO is responsible for implementing the hiring strategy and ensuring that hiring managers follow it and do not violate it.

Hiring for unknown future

Andrew Panfilov — Thu, 02 May 2024 17:49:32 +0000

Originally, this post was published here: https://www.linkedin.com/pulse/hiring-unknown-future-andrew-panfilov-p5hff/

Hiring for a startup presents unique challenges compared to hiring for an established company. Established companies generally have stable operations, products, organizational structures, and cultures. In contrast, startups are dynamic, constantly evolving entities. The organizational structure that was effective six months ago in a startup might already be outdated, necessitating continual adaptation and reorganization. This dynamic nature adds complexity to the hiring process.

A startup's future is often unpredictable and filled with "unknown unknowns." Specialists hired for specific roles may find their skills obsolete or less relevant by the time they are onboarded and ready to contribute due to the fast-paced changes in the startup's needs.

Hiring individuals with previous startup experience can be beneficial. They are likely to be adaptable and experienced in navigating rapidly changing environments. Similarly, diversity in the team can be an advantage, bringing a range of experiences and perspectives that might be critical in unforeseen situations.

Inexperienced but high-potential employees can also be valuable in a startup. Their lack of preconceived notions about what is possible or impossible can lead to innovative solutions to problems that more experienced professionals might find daunting or unsolvable.

A critical aspect of hiring in a startup is ensuring the alignment of values between the founders and potential employees. Skills and knowledge are essential, but if an employee's values don't align with those of the founders, it can lead to significant problems, outweighing the employee's benefits. As such, founders should be deeply involved in hiring, especially in the early stages. This involvement is crucial to maintaining the startup's culture and values, as each management level tends to hire individuals similar to themselves. Misalignment at the higher levels (like C-Suite or VP-Level) can propagate through the organization, potentially leading to internal conflicts that could jeopardize the startup.

For founders, discerning a candidate's calibre is paramount. In a startup's early stages, hiring high-calibre individuals is incredibly crucial. While luck plays a role in success, it cannot be planned or programmed.

Fortunately, the current job market favours employers. A large pool of talented and suitable candidates is available, unlike a few years ago when the market was more candidate-driven.

LLM pipeline for marketing research insights

Andrew Panfilov — Tue, 30 Apr 2024 11:08:10 +0000

Originally this article was published here https://www.linkedin.com/pulse/llm-pipeline-marketing-research-insights-andrew-panfilov-gt1bf/

The past year marked a significant evolution in the marketing research landscape. The advent of widely available consumer-grade Large Language Models (LLMs) has transformed the traditional dichotomy between quantitative and qualitative research methods into a more integrated approach. This new paradigm leverages chatbots to interact with participants, eliciting insights about, for example, their brand preferences. This shift has led to a more nuanced type of research, blurring the lines between quantitative and qualitative methodologies.

High-level representation of Conversational AI pipeline for marketing research:

This innovative approach can be conceptualized as a four-phase process:

Chatbot AI and Survey Integration: This initial phase involves setting up an AI-driven chatbot conversation, sometimes integrated with a traditional survey framework.
- Transition to Phase 2: Once the conversation flow is thoroughly tested, it's published to become accessible to respondents.
Engagement through Conversations: In this phase, the chatbot engages in conversations with respondents.
- Transition to Phase 3: The conversation sessions are concluded once the required respondent data is gathered.
Data Processing and LLM Analysis: This critical stage involves cleansing and analyzing the collected data using LLMs.
- Transition to Phase 4: This phase concludes once the data analysis is complete.
Visual Reporting: The final phase focuses on creating visual reports that effectively communicate the insights derived from the analysis.

In the first stage, researchers develop a finite-state machine to guide the chatbot's conversations. This involves creating a series of prompts, including statements, questions, and follow-up questions used by the LLM during interactions with respondents. In this phase, the researcher employs a method akin to the Read-Eval-Print Loop (REPL). It enables the researcher to observe and assess the chatbot's behaviour in real-time during a conversation, following any modifications made to the dialogue descriptors. This stage is crucial for quality assurance, ensuring that the chatbot's conversations are relevant and error-free. Additionally, the LLM can be utilized for tasks like translating content into various languages.

Respondents engage with the chatbot during the second phase, with each conversation driven by the finite state machine established in the first stage. The dialogue elements are fed into the LLM to generate subsequent statements or questions. A key requirement here is a responsive LLM with low latency (ideally in seconds or less) to prevent long waits for the chatbot's responses. While the OpenAI API may occasionally encounter issues such as 502 and 503 HTTP status codes or connection timeouts, its practical value in production for marketing research remains significant, especially considering that there are no costs associated with failed respondent interactions.

The third stage involves processing the collected chatbot-respondent dialogues with the LLM, following the removal of Personally Identifiable Information (PII). This processing is done in batches to avoid exceeding the LLM's token limits. The LLM's role here is to perform tasks such as categorization, tagging, entity extraction, and sentiment analysis, all essential for deriving meaningful insights from the data. This analysis can be time-consuming, requiring tens of minutes to hours. The system is designed to distribute the dialogues to prevent overloading the LLM API evenly and includes a retry mechanism to handle potential API issues like 5XX HTTP status codes or timeouts. The LLM is also may be employed for validating the results of data aggregation.

In the final stage, the LLM is not utilized. Instead, the aggregated data, now representing valuable marketing insights, is presented in an easily understandable format on a dashboard. This stage focuses on visualizing the insights and offers options to export the data as PDF or PowerPoint reports, facilitating further analysis and presentation.

Components diagram for Phase 1, 3, 4:

Components diagram for Phase 2:

The diagrams illustrate that the LLM is utilized as an opaque entity accessed through an API. In practical applications, especially for marketing research, there is no requirement for fine-tuning the LLM. Currently, gpt-3.5-turbo is the preferred option due to its optimal combination of cost-effectiveness and features. It's important to note that the system represented in these diagrams is a simplified version for the sake of clarity. A real-world implementation would include additional deployable artifacts and specific characteristics. Despite these simplifications, the basic concept remains easily comprehensible.

Need microservice? Take Clojure!

Andrew Panfilov — Fri, 26 Apr 2024 12:49:44 +0000

Initially, this post was published here: https://www.linkedin.com/pulse/lets-write-simple-microservice-clojure-andrew-panfilov-2ghqe/

Intro

This article will explain how to write a simple service in Clojure. The sweet spot of making applications in Clojure is that you can expressively use an entire rich Java ecosystem. Less code, less boilerplate: it is possible to achieve more with less. In this example, I use most of the libraries from the Java world; everything else is a thin Clojure wrapper around Java libraries.

From a business logic standpoint, the microservice calculates math expressions and stores the history of such calculations in the database (there are two HTTP endpoints for that).

Github repository with source code: https://github.com/dzer6/calc

This educational microservice project will provide the following:

Swagger descriptor for REST API with nice Swagger UI console. Nowadays, it is a standard de facto. Microservices should be accessible via HTTP and operate with data in a human-readable JSON format. As a bonus, it is super easy to generate data types and API client code for the client side (it works well for a TypeScript-based front-end, for example).
Postgres-based persistence with a pretty straightforward mapping of SQL queries to Clojure functions. If you have ever used Java with Hibernate ORM for data persistence, you will feel relief after working with the database in Clojure with Hugsql. The model of the persistence layer is much simpler and easier to understand without the need for Session Cache, Application Level Cache and Query Cache. Debugging is straightforward, as opposed to the nightmare of debugging asynchronous actual SQL invocation that is never in the expected location. It is such an incredible experience to see the query invocation result as just a sequence of plain Clojure maps instead of a bag of Java entity proxies.
REPL-friendly development setup. DX (dev experience) might not be the best in class, but it is definitely not bad. Whenever you want to change or add something to the codebase, you start a REPL session in an IDE (in my case, Cursive / IntelliJ Idea). You can run code snippets to print their results, change the codebase, and reload the application. In addition, you can selectively run needed tests. You do not need to restart the JVM instance every time after the codebase changes (JVM is famous for its slow start time). Using the mount library, all stateful resources shut down and initialize correctly every reload.

Leiningen

The project.clj file is a configuration file for Leiningen, a build automation and dependency management tool for Clojure. It specifies the project's metadata, dependencies, paths, and other settings necessary for building the project. Let's break down the libraries listed in the project.clj file into two groups: pure Java libraries and Clojure libraries, and describe each.

Clojure Libraries:

org.clojure/clojure: The Clojure language itself.
org.clojure/core.memoize: Provides memoization capabilities to cache the results of expensive functions.
org.clojure/tools.logging: A simple logging abstraction that allows different logging implementations.
mount: A library for managing state in Clojure applications.
camel-snake-kebab: A library for converting strings (and keywords) between different case formats.
prismatic/schema: A library for structuring and validating Clojure data.
metosin/schema-tools: Utilities for Prismatic Schema.
clj-time: A date and time library for Clojure.
clj-fuzzy: A library for fuzzy matching and string comparison.
slingshot: Provides enhanced try/catch capabilities in Clojure.
ring: A Clojure web applications library.
metosin/compojure-api: A library for building REST APIs with Swagger support.
cprop: A configuration library for Clojure.
com.taoensso/encore: A utility library providing additional Clojure and Java interop facilities.
com.zaxxer/HikariCP: A high-performance JDBC connection pooling library.
com.github.seancorfield/next.jdbc: A modern, idiomatic JDBC library for Clojure.
com.layerware/hugsql-core: A library for defining SQL in Clojure applications.
metosin/jsonista: A fast JSON encoding and decoding library for Clojure.

Pure Java Libraries:

ch.qos.logback: A logging framework.
org.codehaus.janino: A compiler that reads Java expressions, blocks, or source files, and produces Java bytecode.
org.slf4j: A simple logging facade for Java.
org.postgresql/postgresql: The JDBC driver for PostgreSQL.
org.flywaydb: Database migration tool.
com.fasterxml.jackson.core: Libraries for processing JSON.
org.mvel/mvel2: MVFLEX Expression Language (MVEL) is a hybrid dynamic/statically typed, embeddable Expression Language and runtime.

To build the project, just run it in a terminal: lein uberjar

The path to a resulting fat-jar with all needed dependencies: target/app.jar

Frameworks VS Libraries

In the Java world, one common approach is to use full-fledged frameworks that provide comprehensive solutions for various aspects of software development. These frameworks often come with a wide range of features and functionalities built-in, aiming to simplify the development process by providing pre-defined structures and conventions. Examples of such frameworks include the Spring Framework, Java EE (now Jakarta EE), and Hibernate.

On the other hand, in the Clojure world, the approach tends to favour using small, composable libraries rather than monolithic frameworks. Clojure promotes simplicity and flexibility, encouraging developers to choose and combine libraries that best fit their needs. These libraries typically focus on solving one problem well, making them lightweight and easy to understand. Examples of popular Clojure libraries include Ring for web development, Compojure for routing, and Spec for data validation.

The difference between these approaches lies in their philosophies and design principles. Full bloated frameworks in the Java world offer convenience and a one-size-fits-all solution but may come with overhead and complexity. In contrast, small libraries in the Clojure world emphasize simplicity, modularity, and flexibility, allowing developers to build tailored solutions while keeping the codebase lightweight and maintainable.

Docker

Dockerfile sets up a containerized environment for the application, leveraging Amazon Corretto 22 on Alpine Linux. It downloads the AWS OpenTelemetry Agent (you can use the standard one if you don't need AWS-related) to enable observability features, including distributed tracing, and then copies the application JAR file into the container. Environment variables are configured to include the Java agent for instrumentation and allocate 90% of available RAM (which is useful for a container-based setup). Finally, it exposes port 8080 and specifies the command to start the Java application server.

Dev Experience

REPL

The Read-Eval-Print Loop in Clojure is a highly effective tool for interactive development, which allows developers to work more efficiently by providing immediate feedback. Unlike traditional compile-run-debug cycles, the REPL enables developers to evaluate expressions and functions on the fly, experiment with code snippets, and inspect data structures in real time. This makes the development process more dynamic and exploratory, leading to a deeper understanding of the codebase. Additionally, the REPL's seamless integration with the language's functional programming paradigm empowers developers to embrace Clojure's expressive syntax and leverage its powerful features, ultimately enhancing productivity and enabling rapid prototyping and iterative development cycles. REPL is a bee's knees, in other words.

First you start REPL-session:

Next you type (init) to invoke initialization function and press Enter – application will start and you will see something similar to:

The session logs show that the application loads configurations and establishes a connection with a PostgreSQL database. This involves initializing a HikariCP connection pool and Flyway for database migrations. The logs confirm that the database schema validation and migration checks were successful. The startup of the Jetty HTTP server follows, and the server becomes operational and ready to accept requests on the specified port.

After any code change to apply it, you should type (reset) and press Enter.

To run tests, you should type (run-tests) and press Enter.

Docker Compose

Using Docker Compose to run Postgres and any third-party services locally provides a streamlined and consistent development environment. Developers can define services in a docker-compose.yml file, which enables them to configure and launch an entire stack with a single command. In this case, Postgres is encapsulated within a container with predefined configurations. Docker Compose also facilitates easy scaling, updates, and isolation of services, enhancing development efficiency and reducing the setup time for new team members or transitioning between projects. It encapsulates complex configurations, such as Postgres' performance monitoring and logging settings, in a manageable, version-controlled file, simplifying and replicating the service setup across different environments.

Stateful Resources

The mount Clojure library is a lightweight and idiomatic solution for managing application state in Clojure applications. It offers a more straightforward and functional approach than the Spring Framework, which can be more prescriptive and heavy. Mount emphasizes simplicity, making it an excellent fit for the functional programming paradigm without requiring extensive configuration or boilerplate code. This aligns well with Clojure's philosophy, resulting in a more seamless and efficient development experience.

Only two functions: for start and stop.

REST API

Compojure's DSL for web applications makes it easy to set up REST API routes with corresponding HTTP methods. Adding a Swagger API descriptor through libraries like ring-swagger provides a visual interface for interacting with the API and enables client code generation. You can use the Prismatic schema library for HTTP request validation and data coercing to ensure the API consumes and produces data that conforms to predefined schemas. Compojure's middleware approach allows for modular and reusable components that can handle cross-cutting concerns like authentication, logging, and request/response transformations, enhancing the API's scalability and maintainability.

The middleware chain is set up in HTTP server-related namespace:

Developers and QA engineers find Swagger UI console highly convenient. I encourage you to run the service locally and try the console in a browser. Here is a list of HTTP endpoints with data schemas:

Isn't it awesome?

Business Logic

The calc.rpc.controller.calculation controller houses the business logic that defines two primary operations: evaluate and obtain-past-evaluations.

The evaluate operation processes and evaluates mathematical expressions received as requests, storing the results in a database:

The obtain-past-evaluations operation fetches a list of previously executed calculations based on provided offset and limit parameters:

Ensuring that exceptions or database inconsistencies are handled gracefully is crucial for the successful execution of these operations.

The integration of external libraries, MVEL (MVFLEX Expression Language) for expression evaluation, and JDBC for database transactions highlights Clojure's interoperability with Java.

Another essential principle demonstrated by using the MVEL library is never to write your implementation of something already written in Java in Clojure. Most of your business cases are already covered by some Java library written, stabilized, and optimized years ago. You should have strong reasons to write something from scratch in Clojure instead of using a Java analog.

Persistence Layer

Thanks to the hugsql library, we can use autogenerated Clojure functions directly mapped to SQL queries described in a plain text file:

As Clojure is not an object-oriented language, we don't need to specially map query result sets coming from a relational database to a collection of objects in a programming language. No OOP, no ORM. Very convenient. The relational algebra paradigm seamlessly marries with a functional paradigm in Clojure. Very natural:

Compared to NoSQL databases, migrating the data schema in relational databases such as Postgres is a well-established practice. This is typically done through migrations, which is made easy by using the flyway library. To adjust the data schema in Postgres, we simply need to create a new text file containing the Data Definition Language (DDL) commands. In our case there is only one migration file:

Whenever you change an SQL query in the queries.sql file, do not forget to run the (reset) function in the REPL-session console. It automatically regenerates the Clojure namespace with query declarations and runtime-generated SQL wrapper functions.

Configuration

The system uses the Clojure library cprop to manage its configuration. The library adopts a sequential merge policy to construct the application's configuration map. It starts by loading default-config.edn from resources and overlays it with local-config.edn if available. Then, it applies settings from an external config.edn and overrides by environment variables (adhering to the 12-factor app guidelines). This ensures that the latest source has precedence.

The configuration is essential during development and is a Clojure map validated against a Prismatic schema. If discrepancies are detected, the system immediately shuts down, adhering to the fail-fast principle.

Additionally, feature flags within the configuration enable selective feature toggling, aiding in the phased introduction of new functionality and ensuring robustness in production environments.

Logging

The service utilizes org.clojure/tools.logging to offer a logging API at a high level, which works in conjunction with Logback and Slf4j—two Java libraries that are well-known for their reliability in logging. The logging setup is customized for the application's environment: while in development, logs are produced in a plain text format that is easy to read, allowing for efficient debugging. On the other hand, when the service is deployed on servers, logs are structured in a JSON format, which makes them ideal for machine parsing and analysis, optimizing their performance in production.

Tests

This is a real-world industrial example. Yes, we do have tests. Not many. But for this size codebase is pretty much okay.

Unfortunately, most open-source Clojure-based projects on Github do not contain good examples of integration tests. So, here we are, trying to close this gap.

We use the TestContainers library to raise real Postgres instances during the tests. Before Docker and TestContainers, the standard de facto in the Java world was running embedded pure Java database H2, trying to mimic Postgres. It was not good, but there was not much choice then.

The evaluate operation integration test:

The obtain-past-evaluations operation integration test:

After the tests run, you should see this:

Conclusion

Now, when you go through the service codebase and know its internals, you can copy-paste it for yourself, change it according to your requirements, and voila, you will have a really good-looking microservice.

The described codebase is based on years of Clojure programming and a number of projects that have been implemented in Clojure. Some used libraries may look outdated, but in the Clojure world, if a library works, it is okay not to update it often—the language itself is super-stable, and you can easily read and support code written even a decade ago.