DEV Community: Thilanka

Creating custom archetypes in Maven

Thilanka — Sun, 31 Jul 2022 11:15:00 +0000

As developers we embrace concepts like DRY (Don’t Re-invent Yourself). When we are creating similar things repetitively, we always prepare a template for that thing, so instead of starting a project from scratch, you can have a preconfigured directory structure. And in those directories you can place your template source files. Eg: struts.xml, spring config file, web.xml, tiles.xml etc..

Creating your archetype

Run the following command:

mvn archetype:generate \
-DgroupId=[your project's group id] \
-DartifactId=[your project's artifact id] \
-DarchetypeArtifactId=maven-archetype-archetype

This will create following default directory structure for creating archetypes.

archetype
|-- pom.xml
`-- src
    `-- main
        `-- resources
            |-- META-INF
            |   `-- maven
            |       `--archetype.xml
            `-- archetype-resources
                |-- pom.xml
                `-- src
                    |-- main
                    |   `-- java
                    |       `-- App.java
                    `-- test
                        `-- java
                            `-- AppTest.java

Think of maven-archetype-archetype as the maven’s template to create templates.

The generated archetype-descriptor is old and needs to be changed to use new features. To that end, rename the file archetype.xml to archetype-metadata.xml

Over write the following content to the file:

<archetype-descriptor xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.0.0 http://maven.apache.org/xsd/archetype-descriptor-1.0.0.xsd"
   name="simple-web"> <!-- name must be same as the project archetypeId -->
         <fileSets>
         <fileSet filtered="true" encoding="UTF-8">
             <directory>src/main/web</directory>
         </fileSet>
         <fileSet filtered="true" encoding="UTF-8">
             <directory>src/main/resources</directory>
         </fileSet>
         <fileSet filtered="true" encoding="UTF-8">
             <directory>src/test/resources</directory>
         </fileSet>
         <fileSet filtered="true" packaged="false">
             <directory>src/main/java</directory>
             <includes>
                 <include>**/*.java</include>
             </includes>
         </fileSet>
         <fileSet filtered="true">
             <directory>src/test/java</directory>
             <includes>
                 <include>**/*.java</include>
             </includes>
         </fileSet>
     </fileSets>
 </archetype-descriptor>

The above maven archetype descriptor file tells maven to copy (or create if not exists) any source files reside in web, src/main/java and corresponding test source directories. The archetype-resources directory is where all your template code should go. When you finally copy all your template files run the following command in maven to install your newly created artefact.

mvn install

You might want to continuously improve your template over time. Your artefact get installed multiple times during the process, so wrong versions of archetype may be installed. To counter that you might want to clean archetype cache by issuing following command:

mvn clean install -U

-U means force update of dependencies.

So whenever you want to create a new project using your archetype run following command:

mvn archetype:generate -DarchetypeGroupId=com.my-template
 -DarchetypeArtifactId=template-name -DarchetypeVersion=1.0
 -DgroupId=proj.groupid -DartifactId=projectid

So there you have it! This single line command can load all your template source files and create whole directory structure of your application.

Sine wave generator using Golang

Thilanka — Wed, 20 Jul 2022 13:09:54 +0000

Let's build a continuous sine wave generator with Go. This could be the first step into creating your ground breaking synthesizer, who knows :) But some theory first before we dive in.

How digital audio works

We know that sound waves are vibrations travelling through air. These vibrations cause air pressure changes from sound source (Generator in our case) to receiver (our ears) over time. It is important to note that air particles do not travel with vibrations to receiver. So fundamentally the air pressure and the time is only needed to capture sound from the receiver's perspective.

Number of pressure variations per second is called the frequency of sound. The size of the pressure difference is called the amplitude of the sound. Silence is when the sound pressure remains at the atmospheric pressure.

Sample rate and Nyquist theorem

In order to represent sounds in the digital world, we need to take samples of sound pressure levels at a regular intervals. Number of samples taken per second is known as the sample rate. A higher sample rate can accurately represent an audio signal at the cost of increased processing power and memory while a lower sample rate may miss higher frequencies. So it's clear that sample rate depends on the frequency of the signal. Since human hearing is limited to 20hz to 20Khz frequency range, our sample rate should support this range. But how do we know the optimum sample rate for reproducing an audio signal in our audible frequency spectrum? Well Nyquist–Shannon sampling theorem states that it's possible to reproduce any sound if the sampling rate is two times the highest frequency of the source signal. Since most humans cannot listen to frequencies above 20khz, a sample rate of 40khz is theoretically sufficient. But the CD quality standard is defined as 44.1kHz/44000Hz.

What we are going to build?

We are going to build a sound wave generator that's going to emit a sine wave with a given frequency continuously in Go.

We will be using beep go package to process audio signals and playback sounds. Beep uses oto under the hood for audio playback.

Setting up project

Create a new directory and initialize root module

go mod init github.com/nuwan89/sine-generator

This will create the root module as follows:

module github.com/nuwan89/sine-generator

go 1.17

Installing Go dependencies

go get github.com/faiface/beep 
go get github.com/hajimehoshi/oto

Initializing beep

Stick the below code inside the main function. Don't worry we'll go through each line.

func main() {
    sr := beep.SampleRate(44100)
    speaker.Init(sr, sr.N(time.Second/10)) // sr.N(time.Second/10) = buffer size for duration 1/10 second
    sine, _ := SineTone(sr, 100)
    speaker.Play(sine)
        select {} // makes the program hang forever
}

Firstly we define the sample rate as 44100Hz.
Then we initialize the Speaker with the sample rate and the buffer size (one tenth of a second).
- Instead of reading samples in real-time, the speaker reads from a buffer with a delay. This allows the audio to be played back in a continuous manner without drop outs and glitches.
SineTone is a builder function for a Streamer that we are going to implement in which its only job is to stream infinite sequence of audio samples. It needs a sample rate and a frequency (100Hz)
Finally we can blast audio through the speakers using the given Streamer :)

type SineWave struct {
    sampleFactor float64 // Just for ease of use so that we don't have to calculate every sample
    phase        float64
}

func (g *SineWave) Stream(samples [][2]float64) (n int, ok bool) {
    for i := range samples { // increment = ((2 * PI) / SampleRate) * freq
        v := math.Sin(g.phase * 2.0 * math.Pi) // period of the wave is thus defined as: 2 * PI.
        samples[i][0] = v
        samples[i][1] = v
        _, g.phase = math.Modf(g.phase + g.sampleFactor)
    }

    return len(samples), true
}

func (*SineWave) Err() error {
    return nil
}

SineWave is just a struct which stores properties of our sine wave. Phase of the sound wave is basically the current position of the sine wave. This is represented by radians.
Stream function is the most important one here. It receives a two-dimensional array which we are going to fill. Each dimension represents left and right channel of the signal.
We are looping through each sample point and calculate the sine value of the phase.
We need to factor in frequency for calculating the next phase.
math.Modf gives us the fractional component so that the phase cycles between 0 - 1

I have copied the SineTone initializer method for the sake of completeness.


func SineTone(sr beep.SampleRate, freq float64) (beep.Streamer, error) {
    dt := freq / float64(sr)

    if dt >= 1.0/2.0 {
        return nil, errors.New("samplerate must be at least 2 times grater then frequency")
    }

    return &SineWave{dt, 0.1}, nil
}

Oh and one more thing.. You need to run tidy command to add import statements in your module file:

go mod tidy

Finally you can run it using:

go run main.go

Enjoy the never ending 100Hz sine wave awesomeness! :)

GitHub project

Choreography Saga Pattern Implementation

Thilanka — Thu, 23 Jun 2022 17:32:28 +0000

Saga pattern can be applied to perform distributed transactions spanning across multiple microservices by maintaining an application level locking mechanism.

In a traditional monolith, a single database performs all the application wide transactions so it can use table/row level locking to enforce ACID compliance. Microservices based systems usually have a database per service so that this ACID compliance could not be achieved in the database layer.

Saga pattern can be implemented in two ways:

1. Choreography/Event based Saga

Much like how ballet dancers perform set of precise steps in coordination with each other to the sound of music, each microservice participate in the transaction emits events that other microservices can capture and act accordingly. For eg:

A user submits an order through a website
NEW_ORDER event will be created.
Accounts microservice will deduct the order amount from the customer account and emit PAYEMENT_RECEIVED event.
Inventory microservice can act on the event and adjust the inventory.
When the INVENTORY_RESERVED emitted, the delivery microservice can schedule the delivery
Finally order created website will display the delivery details to the user.

But what happens when the inventory microservice rejects or fails the request to reserve a particular item simply because it is not available? Since the account microservice has already deducted the amount from the user's account and it's too late for a database level rollback, a compensating transaction has to be performed in the Accounts microservice. (repayment of the deducted amount) Again such error/failure scenarios are communicated to others by the same event emitting pattern.

Choreography best fits in event-driven communication

2. Orchestrator based Saga

In a musical orchestra, a conductor is a musical professional who directs the performance with movements of hands. Conductor sets the tempo and unifies the orchestra by providing instructions to the fellow musicians.

Orchestrator service in the Saga pattern acts as the conductor. The musicians who act on the instructions/commands received by the conductor are the microservices participate in the distributed transaction. Unlike choreography pattern where microservices act upon other microservices' events, Orchestrator based Sagas require a separate module/service known as the Saga Orchestrator to control the flow of the transaction.

Once a request has been posted to initiate a distributed transaction to the Saga orchestrator, it commands each microservice in a predefined sequence to do some task related to the transaction. Microservices perform their tasks and publish results typically into a single reply channel. Based on the events gathered in the reply channel, the orchestrator can decide whether to continue the transaction or execute compensating transactions on the corresponding microservices.

Let's translate the above ordering system to Orchestrator pattern:

A user submits an order through a website and sends a request/command to initiate Order Saga Orchestrator(OSO)
OSO commands Accounts microservice to deduct the order amount from the customer account
If the account has enough funds, Accounts microservice emits PAYEMENT_RECEIVED event into the reply channel.
OSO commands Inventory microservice to reserve inventory.
If the inventory is available it will emit INVENTORY_RESERVED event into the reply channel.
If however OSO receives an INVENTORY_REJECTED, it will command the Accounts microservice to execute compensating transaction (repayment of the deducted amount)

Orchestrator best fits in command-driven communication

Implementing Choreography Saga

We will be implementing the above discussed scenario with a Spring Boot project. RabbitMQ's Topic Exchange will be utilized for routing messages for multiple queues via a routing key.

A quick refresher on RabbitMQ

RabbitMQ is an opensource message broker which allows storing messages in queues. We will be using three queues and an Exchange. Message producers send messages to exchanges and exchanges route messages to queues so that they can be consumed by consumers.

How does the exchange knows the destination queue? Well each message has a property called a routing key embedded in the header. When a queue is created, it will be bound to the exchange by the routing key so that exchange knows where a given message should end up.

A picture is worth a thousand words:

An exchange is known as a Topic Exchange when it's bound to many queues using a routing key. This relationship is depicted in the highlighted yellow boxes in the diagram.

Project setup overview

We will be creating three Spring Boot projects for each microservice:

Order Service
- Creates a new order
- Listens to status updates of the order
Accounts Service
- Checks if the account has enough funds
- Debits account on success
- Credits account on compensating transaction request
- Publishes order status failure
Inventory Service
- Checks if inventory item requested is available
- Adjust inventory on success
- Requests account service for compensating transaction
- Publishes status failure

Dependencies



<!-- All microservices' pom.xml -->
<dependency>             
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

...

<dependencyManagement>
    <dependencies>
        <dependency>         
            <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-dependencies</artifactId>
        <version>${spring-cloud.version}</version>
        <type>pom</type>
        <scope>import</scope>
    </dependency>
    </dependencies>
</dependencyManagement>

Configuring RabbitMQ

RabbitMQ Beans definition

All Microservices declare queues and bindings so that queues and exchanges readily available regardless of the microservices' start order. Check code comments to get a better understanding.



@Configuration
public class RabbitBeanConfig {
    public static final String SINGLE_ORDER_TOPIC_EXCH_NAME = "single-orders-exchange";
    public static final String SINGLE_ORDER_QUEUE_NAME = "single-order-queue";
    public static final String SINGLE_ORDER_ROUTING_KEY = "orders.create.*";

    @Bean
    Queue singleOrderqueue() {
        return new Queue(SINGLE_ORDER_QUEUE_NAME, false); // New orders received at our Restful endpoint will be placed on this queue to be consumed by Accounts microservice
    }

    @Bean
    Queue orderUpdateQueue() {
        return new Queue("order-update", false); // Stores the final order outcome - Order Manager microservice will consume this
    }

    @Bean
    Queue paymentQueue() {
        return new Queue("payment", false); // Successful order payments end up here to be consumed by Inventory service
    }

    @Bean
    TopicExchange singleOrderExchange() {
        return new TopicExchange(SINGLE_ORDER_TOPIC_EXCH_NAME); //Single Topic Exchange to route messages to queues using a Routing key
    }

    @Bean
    Binding singleOrderBinding(Queue singleOrderqueue, TopicExchange singleOrderExchange) {
        return BindingBuilder.bind(singleOrderqueue).to(singleOrderExchange).with(SINGLE_ORDER_ROUTING_KEY); // Bind each queue to the Exchange
    }

    @Bean
    Binding orderUpdateBinding(Queue orderUpdateQueue, TopicExchange singleOrderExchange) {
        return BindingBuilder.bind(orderUpdateQueue).to(singleOrderExchange).with("orders.update");
    }

    @Bean
    Binding inventoryBinding(Queue paymentQueue, TopicExchange singleOrderExchange) {
        return BindingBuilder.bind(paymentQueue).to(singleOrderExchange).with("orders.pay");
    }

    @Bean
    MessageConverter messageConverter() { // Encode/Decode Java objects to/from JSON 
        return new Jackson2JsonMessageConverter();
    }

    @Bean
    RabbitTemplate rabbitTemplate(final ConnectionFactory connectionFactory, final MessageConverter messageConverter)
    {
        final RabbitTemplate rabbitTemplate = new RabbitTemplate();
        rabbitTemplate.setExchange(SINGLE_ORDER_TOPIC_EXCH_NAME); //Otherwise default rabbit queue
        rabbitTemplate.setConnectionFactory(connectionFactory);
        rabbitTemplate.setMessageConverter(messageConverter); // Set the message converter to the template
        return rabbitTemplate;
    }

}

Queue listeners

Order Manager needs to listen to order-update queue in which contains the final status of the transaction.



@Component
public class OrderUpdateListener {

    @Autowired
    Queue orderUpdateQueue;

    @RabbitListener(queues = "#{orderUpdateQueue.getName()}")
    public void onOrderUpdate(SingleOrder order) {
        System.out.println("ORDER_SERVICE|"+order.id+"|ORDER STATUS UPDATED: "+order);
    }
}

SingleOrder is a simple DTO object representing Saga transaction details.



public class SingleOrder implements Serializable {

    public int id;
    public String item;
    public int qty;
    public int cost;
    private int costFactor = 10;
    public boolean isFailed;
    public String failureReason;

    public SingleOrder() { }

    public SingleOrder(int id, String item, int qty) {
        this.id = id;
        this.item = item;
        this.qty = qty;
        this.cost = qty * costFactor;
    }
}

Accounts service and the Inventory service has listeners configured as well. When a new Order is created Accounts service should check if there are enough funds to do the transaction. It should also support compensating transactions when Inventory service rejects the transaction.



@Component
public class OrderCreatedListener {

    @Autowired
    Queue createOrderQueue;

    @Autowired
    RabbitTemplate rabbit;

    @RabbitListener(queues = "#{createOrderQueue.getName()}")
    public void onOrderCreated(SingleOrder order, @Header(AmqpHeaders.RECEIVED_ROUTING_KEY) String key) {
        System.out.println("ACCOUNT_SERVICE|"+order.id+"|RESPONDING TO ORDER CREATED  key: "+key);
        String[] tokens = key.split("\\.");
        if (tokens.length > 1 && tokens[tokens.length - 1].equals("failed")) {
            //Compensating transaction
            System.out.println("ACCOUNT_SERVICE|"+order.id+"|COMPENSATING AMOUNT = +"+order.cost);
        } else {
            if (order.cost > 100) {
                System.out.println("ACCOUNT_SERVICE|"+order.id+"|NOT ENOUGH FUNDS");
                order.isFailed = true;
                order.failureReason = "NOT ENOUGH FUNDS: " + order.cost;
                rabbit.convertAndSend("orders.update", order);
            } else {
                System.out.println("ACCOUNT_SERVICE|"+order.id+"|DEBITED AMOUNT = -"+order.cost);
                rabbit.convertAndSend("orders.pay", order);
            }
        }
    }
}

Similarly the Inventory service listens to the verified payments queue for accepting/rejecting the transaction. If the item is not available it should send a message to order-update queue so that the Order Manager can show rejected order details to the customer. It should also notify Account Manager to run a compensating transaction.



@Component
public class PaymentReceivedListener {

    @Autowired
    Queue paymentQueue;

    @Autowired
    RabbitTemplate template;

    @RabbitListener(queues = "#{paymentQueue.getName()}")
    public void reserveInventory(SingleOrder order) {
        System.out.println("INVENTORY_SERVICE|"+order.id+"|VERIFIED PAYMENT");
        if (order.qty > 5) {
            System.out.println("INVENTORY_SERVICE|"+order.id+"|STOCKS NOT AVAILABLE");
            order.isFailed = true;
            order.failureReason = "STOCKS NOT AVAILABLE: " + order.qty;
            template.convertAndSend("orders.create.failed", order);
            template.convertAndSend("orders.update", order);
        } else {
            System.out.println("INVENTORY_SERVICE|"+order.id+"|STOCKS RESERVED: QTY" + order.qty);
            template.convertAndSend("orders.update", order);
        }
    }
}

We are almost done. Our simple order creation restful web endpoint would look like as follows.



@RestController
@RequestMapping("/single-order")
public class SingleOrderResource {

    @Autowired
    RabbitTemplate rabbitTemplate;

    @GetMapping()
    public SingleOrder createSingleOrder(@RequestParam String item, @RequestParam int qty) {
        int id  = ThreadLocalRandom.current().nextInt(100, 500 + 1);
        SingleOrder order = new SingleOrder(id, item, qty);
        System.out.println("ORDER_SERVICE|NEW ORDER:"+order);
        rabbitTemplate.convertAndSend("orders.create.new", order);
        return order;
    }
}

That's pretty much how you would implement a choreography based Saga :) If you have any questions please do let me know in the comment's section.

Remove unused CSS from Svelte

Thilanka — Fri, 23 Jul 2021 06:08:49 +0000

When I included Bulma CSS my css bundle size increased up to 1MB, which is quite large for a website. After purging unused css I was able to reduce the bundle size close to 140KBs which is much more acceptable. In this post I will describe the way I did it :)

Adding required dependencies

npm i @fullhuman/postcss-purgecss postcss-load-config

Create the postcss.config.js file at the root of your project. This will instruct postcss-purgecss how to do its job.

const purgecss = require("@fullhuman/postcss-purgecss")({
    content: ["./public/**/*.html", "**/**/*.svelte"], //filtering all svelte
    whitelistPatterns: [/svelte-/, /fa-icon/], //Svelte compiler adds these prefixes. We won't mess with it!
    defaultExtractor: (content) => content.match(/[A-Za-z0-9-_:/]+/g) || [],
})
const dev = process.env.ROLLUP_WATCH
module.exports = {
    plugins: [...(!dev ? [purgecss] : [])], //We are only purging in prod builds
}

The above postcss.config.js file will be loaded automatically by the postcss-load-config plugin. There we include all svelte files and index.html file to purge css plugin so it can analyze the css usage and remove unused css from the bundle. We want to keep all css classes that the Svelte compiler prefixes so we can include it in the whitelistPatterns option. Rest is self explanatory.

Finally it’s time to tell rollup to incorporate purgecss plugin using svelte-preprocess.

...
svelte({
    preprocess: sveltePreprocess({
        defaults: {
            style: 'scss',
        },
         scss: {
            //Customize bulma colors
            prependData: `@import 'src/styles/variables.scss';`, 
},
postcss: production // To enable purging unused css
}),

We are loading Bulma CSS using the variables.scss file and we enable postcss in the sveltePreprocess plugin in production mode. So that’s pretty much it!