DEV Community: Carlos Azuaje

I created a Monorepo Chat architecture solution

Carlos Azuaje — Sun, 26 Nov 2023 16:00:11 +0000

Realtime Chat using Apache Camel, Cassandra and Firestore

Source Code:

https://github.com/CharlyJazz/Chat-Firebase-Apache-Camel

If you one learn about Python, Microservices, Next JS, GCP, Kubernetes you are welcome to check it out the code

Chat system architecture leveraging Apache Camel's aggregation pattern. This architecture incorporates a mechanism to buffer incoming messages for a specific timeout duration, measured in seconds. Subsequently, the system aggregates these buffered messages based on their unique chat IDs. The purpose of this aggregation step is to optimize Firestore usage by reducing the number of write operations. The aggregated message data is then efficiently dispatched to Firestore, with the primary goal of minimizing Firestore billing costs.

The chat pagination utilize Cassandra instead of Firestore. This strategic decision serves a dual purpose:

It alleviates the load on Firestore's read operations
Harnesses the inherent advantages of Cassandra for optimizing the partitioning using the chat_id as the partition key.

The next content of this paper explain the sistem using a Architectural Map Features approach.

Problem

Create a chat that makes spam difficult, optimizes availability, and manages reading and writing operations in Firestore to reduce expenses.

The Features are:

Create User
Login
Get the list of users
User send messages to other user
Create a new chat room between to users: The user A will find the user B and start a chat.
User get realtime message updating

Mapping Features to Architectural Components

Feature Mapping 1: The users management and authentication (features 1, 2 and 3) will be handle by a microservice called Auth Microservice
- The microservice will have a store.
- The microservice will use JWT for the Authentication.
Feature Mapping 2. Chat creation and Message creation/retrieving (features 4 and 5) will be handle by a microservice called Chat Microservice
- The microservice will have a store.
Feature Mapping 3. Chat Microservice will take care of sending the messages to a queue proccesing in other to satisfy the feature 6

Realtime solution to satisfy feature 6

Instead of using Firestore or another cloud-based solution for a complete chat solution, we are going to use Firestore only to update the UI with the aggregated news messages.

The aggregated news messages refers to a set of messages sent by a user. Instead of displaying each message one by one, there will be a server that will group all the related messages sent by a user into a list. This list will then be retrieved for the chat..

This is how it looks like in Firestore:

Architectural Components to explicit implementation details

The auth and chat microservices will be written in Python using FastAPI.
The message aggregating process will be handled by Apache Camel using SpringBoot on Java.
User databases will be Postgres.
Chat database will be Cassandra to create a fast partition for each chat and fast pagination based on Time UUID.
The aggregated news messages will be saved into Firestore and only will retrieve for realtime
Chat messages queue processing will be handle by Kafka using a Python Kafka Consumer.
Auth validation in Python microservices will be handled by a Python shared library called auth-validator

Chat messages pagination and retrieving of the first page

The Chat will send a HTTP request to the Chat Microserver /chat/<id>?time=<time-uuid>&quantity to retrieve a set of messages

Query parameter quatity will be a integer to know the max size of messages to get
Query parameter time will be the UUID time of the last message that the chat have to get messages before that time
Both query parameter are optional. When there are not quantity you get 10. When there are not time you get the latest messages.

Development Enviroment Details

We going to use tilt in order to run the infraestructure on Kubernetes for development purposes.
There is a Docker Compose for development purposes.
For Python you need Pipenv and version 3.9.7
For Java you need:

java 17.0.2 2022-01-18 LTS
Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)

Running the project using Docker Compose:

docker-compose rm -svf && docker-compose up

It should create 9 containers:

1.) chat-firebase-apache-camel_zookeeper_1
2.) chat-firebase-apache-camel_cassandra_1
3.) chat-firebase-apache-camel_auth_db_1
4.) chat-firebase-apache-camel_kafka_1
5.) chat-firebase-apache-camel_consumer_1
6.) chat-firebase-apache-camel_auth_microservice_1
7.) chat-firebase-apache-camel_apache_camel_microservices_1
8.) chat-firebase-apache-camel_chat_microservices_1
9.) front_end_next_js_1

Then you can open http://localhost:3000/ and start using the chat.

Run tests in Apache Camel

mvn test -X

How to run the microservices manually?

Chat Microservice:

cd chat-microservice && pipenv run server

Auth Microservice:

cd auth-microservice && pipenv run server

You will need Maven to run the Apache Camel Spring Boot server

cd apache-camel-service-bus && mvn package && mvn spring-boot:run

Docker Development Worflow

Main command to run docker-compose

docker-compose rm -svf && docker-compose up

Migrate using Alembic in the Auth Microservice

In order to use the auth microservice you must migrate using alembic to create the user table. Access to the container and run:

pipenv shell
alembic upgrade head

It should print something like:

dict_keys(['user'])
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> fe1a63894533, create user table
INFO  [alembic.runtime.migration] Running upgrade fe1a63894533 -> d3c6204b9f3c, username unique

Run Kafka CLI:

Create a topic:

docker exec -it CONTAINER_ID kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic charlytest

List of topics:

docker exec -it CONTAINER_ID kafka-topics.sh --bootstrap-server localhost:9092 --list

Get topics information

docker exec -it CONTAINER_ID kafka-topics.sh --bootstrap-server localhost:9092 --describe

Console Producer using key and acks

In this case 12345 is the key and hello de message content (Useful to put all the messages in the same partition using the user_id)

docker exec -it CONTAINER_ID kafka-console-producer.sh  --bootstrap-server localhost:9092 --topic charlytest --producer-property acks=all --property parse.key=true --property key.separator=:

>> 12345:hello

Access to Cassandra:

docker exec -it CONTAINER_ID cqlsh

Get the network information

docker network ls | grep "camel"

Firebase integration for Firestore.

You need to download the service_account_key.json from your Firebase project add it to the aggregated-messages-consumer folder.

Also you need to update the .env file in the frontend-dev folder to looks like:

# MICROSERVICES
NEXT_PUBLIC_AUTH_MICROSERVICE = 'http://0.0.0.0:8000'
NEXT_PUBLIC_CHAT_MICROSERVICE = 'http://0.0.0.0:9000'
# FIREBASE
NEXT_PUBLIC_API_KEY="..."
NEXT_PUBLIC_AUTH_DOMAIN="..."
NEXT_PUBLIC_PROJECT_ID="..."
NEXT_PUBLIC_STORAGE_BUCKET="..."
NEXT_PUBLIC_MESSAGING_SENDER_ID="..."
NEXT_PUBLIC_APP_ID="..."

License

Apache License 2.0

I wrote a super hexagon clone game in python pygame

Carlos Azuaje — Sun, 25 Oct 2020 23:51:23 +0000

If someone want learn how it work check the code:

https://github.com/CharlyJazz/Hexagonal-Game

Feedback welcome!

Answer: Decimal or numeric values in regular expression validation

Carlos Azuaje — Sat, 06 Jun 2020 00:52:44 +0000

answer re: Decimal or numeric values in regular expression validation

Sep 8 '16

Actually, none of the given answers are fully cover the request.
As the OP didn't provided a specific use case or types of numbers, I will try to cover all possible cases and permutations.

Regular Numbers

Whole Positive

This number is usually called unsigned integer, but you can also…

Open Full Answer

Experience building a package from react to svelte

Carlos Azuaje — Fri, 15 May 2020 21:44:15 +0000

The svelte ecosystem grows more and more. New packages appear and people are encouraged to start using it.

I decided to contribute creating a package that perhaps will be useful to some people.

The original react project is this, a component to render a credit card. Very cool.

And this was my experience:

1.) Folder structure:

I had no idea how to create an npm package for svelte. So dig a little. In the github profile of sveltejs i founded this
A template to create shareables components.

2.) Sandbox environment to test the svelte component

My first idea was Storybook.

The new packages:

    "@babel/core": "^7.9.6",
    "@storybook/addon-actions": "^5.3.17",
    "@storybook/addon-links": "^5.3.17",
    "@storybook/addons": "^5.3.17",
    "@storybook/svelte": "^5.3.18",
    "babel-loader": "^8.1.0",

And a config file:

module.exports = {
  stories: ["../src/*.stories.js", "../examples/*.stories.js"],
  addons: ["@storybook/addon-actions", "@storybook/addon-links"]
};

What scenarios should we try with Storybook?

A file answered this question for me, the unit tests.

So I transformed the test cases into a storybook:

import CreditCard from "./";

export default {
  title: "Credit Card",
  component: CreditCard,
};

export const AmericanExpress = () => ({
  ...
});

export const Dankort = () => ({
  ...
});

// Diners, Visa, Maestro, ETC...
});

3.) CSS y JSX:

This stage seemed complex. Since it involved the adaptation of CSS and React JSX to Svelte. I didn't know what bugs was going to find me with.

Browsing the original react package repository i discovered this A good SCSS file with all styles. I didn't think twice and put it inside the style tags in our new svelte component.

It worked. Now the JSX

Copy and paste the JSX to Svelte and replace className with class.

it worked again!

4.) Javascript, the challenge:

The first step was to write the script tags in the svelte component.

 // This will be insane

First we had to create the props.

  export let focused = "number";
  export let acceptedCards = [];
  export let cvc = "";
  export let expiry = "";
  export let issuer = "";
  export let name = "";
  export let number = "";
  export let locale = { valid: "VALID THRU" };
  export let placeholders = { name: "YOUR NAME HERE" };
  export let preview = false;

Now we need usepayment as in the React project to manage credit cards and their validations. And we needed to implement a method in the constructor like in the version de react. The closest thing is onMount in Svelte.

onMount(() => {
  // Logic for credit cards
});

It worked.

Now we need derived data (Format credit number from the text props, format the CVC with the slash MM/YY). As we can see the react component used get to create classes methods that don't need be called like a function call_me_i_am_a_function() and the behavior of this method is like a property. Very common in object oriented programming:

 get number() {
    const { number, preview } = this.props;

    let maxLength = preview ? 19 : this.options.maxLength;
    let nextNumber = typeof number === 'number' ? number.toString() : number.replace(/[A-Za-z]| /g, '');
...

So it seemed impossible to use something like that in Svelte. It is time to research.

The weird dollar symbol. What?

Lets try:

$: expiryDerived = (() => {
    focused = "number";
    const date = typeof expiry === "number" ? expiry.toString() : expiry;
    let month = "";
    // .... bla bla bla
    return value
  })();

it worked, wow. At this time I am not sure if what I am doing is a good practice. If any of you have any ideas please submit a pull request: star:

5.) A bug when compiling.

It seems that payment uses commonjs and we don't have a plugin for this in our rollup configuration that gave us the svelte template.

So I had to add the commonjs plugin for rollup.

import commonjs from '@rollup/plugin-commonjs';

plugins: [
  svelte(),
  resolve(),
  commonjs() // <-- Our new friend
]

Easy, right?

6.) Publish NPM Package

npm publish

Conclusion:

Everything very easy and fun. There was not much stress. Svelte fulfilled all the requirements to create this component.

Enhacements

Refactor CSS to make customizable
Delete the payment package because it is very old and there is a news alternatives to manage credit card validation
Use Svelte animation and not @keyframes
Add unit test

Contributions

Send a pull requests to https://github.com/CharlyJazz/svelte-credit-card