DEV Community: Felicia Faye

Resilient microservices with Quarkus and Kotlin - All threads together

Felicia Faye — Mon, 25 Jul 2022 10:13:47 +0000

The second part of this series was all about theory - we had a closer look at some helpful resilience design patterns. Now it's time to take some action 💪. Let's finally bring all threads together and see how we can realize this with Quarkus and Kotlin.

Do you still remember the shocking scenario for all of this? We are doomed cause we live in an area with poisonous snakes and must rely on our own service to be warned about the dangerous animals in time.

Let me introduce you to the Snake Alarm Detection Fulfilment Scenario where we are going to handle this in a resilient way.

Imagine small snake detector boxes, which can magically detect snakes passing by. Sometimes the detectors have false positives with cute and harmless lizards. To handle these false positives the snake detector boxes can also send a photo of the detected animal via REST endpoint so it can be evaluated, for example, by means of machine learning. When we are sure that it's a snake, we want to send a push notifications to all mobile devices in that area. Sadly, we have to use an external service for the push notifications and this is pretty expensive. We have two options here: one service, which is very expensive and 100% reliable but a little bit slow and another services, which is much cheaper and faster, but not so reliable at all.

The plan is to build a robust roundtrip from a) detecting a "snake or lizard" with the box to b) sending a warning push notification to everyone in the area. Ideally the push notification goes out fast and we do not spend that much money on sending those push notifications.

Let's try: the snake detector box could make a POST request to a SnakeAlarmEndpoint. This could trigger a SnakeAlarmRepository to emit an event into an AMQP messaging channel named snake-alarm-out. Then we subscribe to those events in another snake-alarm-in channel to be able to scale independently. Given the illustration below, you can see our SnakeAlarmProcessor, which can process new SnakeAlarms coming in from the AMQP channel snake-alarms-in.

The SnakeAlarmProcessor itself calls the LizardCheck to verify that we have a snake and not a lizard coming in.
The LizardCheck has a runtime between 0,2s and 1,06s. Having the runtime measured, we can configure our SnakeAlarmProcessor call to the LizardCheck with a timeout of 1s and a maximum of 3 retries to make most calls succeed in an acceptable time.

Finally we call the push notification service. The primary service of choice will be tried first, but fails with a probability of 50% and has an average response time of 1s. We give that a maximum of 2 retries. After that we call the secondary service of choice as a fallback, which has a probability of ~100% and a response time of 3s.

Sample output of the running result
Here you find the output of the final result. Try to see where we had a false positive and where not. Where did we use the expensive push notification service? It's all written in the logs.

2022-07-24 16:03:53,936 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:53,994 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,000 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=1, last=1, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,001 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=1, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,460 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,463 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=2, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,464 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=2, last=2, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,579 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:54,937 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:56,180 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,185 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=3, last=3, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:56,186 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=3, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,618 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,622 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=4, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,625 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=4, last=4, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,105 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,110 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=5, last=5, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,111 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=5, deliveryTag=\x02, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:57,219 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:57,534 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,538 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=6, last=6, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,539 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=6, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:58,039 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:58,041 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], Danger - Successfully verified a snake
2022-07-24 16:03:58,052 INFO  [de.fay.sna.PushNotificationService] (vert.x-eventloop-thread-0) Push notification estimated in 3 sec
2022-07-24 16:03:58,420 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard

Postman
This is an exemplary POST request we are going to use to fire up our services. If we make the request right, we should receive a 202 Accepted response. You will find this as a Postman collection in the Github sources.

Snake alarm endpoint
Our endpoint calls the SnakeAlarmRepository to take care of everything.

@Path("snake-alarm")
class SnakeAlarmEndpoint(
    private val snakeAlarmRepository: SnakeAlarmRepository
) {
    @POST
    @Consumes(MediaType.APPLICATION_JSON)
    @Produces(MediaType.TEXT_PLAIN)
    suspend fun snakeAlarm(request: SnakeAlarmRequest): Response {
        return when (snakeAlarmRepository.alarm(request.snakes)) {
            true  -> Response.accepted().build()
            false -> Response.serverError().build()
        }
    }
}

The SnakeAlarmRepository emits the alarm without verification using an AMQP messaging channel named snake-alarms-out.

@ApplicationScoped
class SnakeAlarmRepository(
    private val pushNotificationService: PushNotificationService,
    @Channel("snake-alarms-out") private val snakeAlarmEmitter: Emitter<SnakeAlarm>,
    private val logger: Logger
) {
    suspend fun alarm(items: List<SnakeItems>): Boolean {
        val snakeAlarm = SnakeAlarm(
            snakes = items,
            snakeVerification = "Snake could also be a lizard :-)",
            estimatedPushNotificationTimestamp = null
        )
        return try {
            snakeAlarmEmitter.send(snakeAlarm).await()
            true
        } catch (e: Exception) {
            logger.log(Level.SEVERE, "Error while sending snake alarm", e)
            false
        }
    }
}

Application Properties
That's where we need to configure our loosely coupled AMPQ messaging channels, which will run using a Docker image (a sample Docker file is provided in the Github sources).

quarkus.log.level=DEBUG
quarkus.log.category."org.jboss.resteasy".min-level=DEBUG
# The AMQP broker location and credentials
amqp-host=localhost
amqp-port=5672
amqp-username=quarkus
amqp-password=quarkus
# Configure snake alarm messaging
mp.messaging.incoming.snake-alarms-in.connector=smallrye-amqp
mp.messaging.incoming.snake-alarms-in.address=snake-alarms
mp.messaging.outgoing.snake-alarms-out.connector=smallrye-amqp
mp.messaging.outgoing.snake-alarms-out.address=snake-alarms

When we receive a message from this channel, the SnakeAlarmProcessor checks if it really is a snake by using the LizardCheck.

@ApplicationScoped
class SnakeAlarmProcessor(
    private val pushNotificationService: PushNotificationService,
    private val lizardCheck: LizardCheck,
    private val logger: Logger
) {
    @Incoming("snake-alarms-in")
    suspend fun processSnakeAlarm(snakeAlarm: SnakeAlarm) {
        val isVerifiedSnake = lizardCheck.checkSnakeAlarmValid(snakeAlarm)
        val verificationResultText = if (isVerifiedSnake) "Danger - Successfully verified a snake" else "False positive - Successfully verified a lizard"

        logger.info { "Received an alarm for snake alarm fulfillment: ${snakeAlarm.snakes}, $verificationResultText" }
        if(isVerifiedSnake) pushNotificationService.announcePrimaryPushNotificationService()
    }
}

The LizardCheck has often timeouts. Therefor we have configured timeout and retry options by using annotations.

@ApplicationScoped
class LizardCheck {
    @Timeout(1, unit = ChronoUnit.SECONDS)
    @Retry(maxRetries = 3)
    suspend fun checkSnakeAlarmValid(): Boolean {
        if (Random.nextBoolean()) {
            delay(Random.nextLong(200, 1060))
            return false
        }
        return true
    }
}

The PushNotificationServices has a method to announce each notification services. The first method has the second method has a fallback. If the fallback is used, we have a max retry of two attempts for that.

@ApplicationScoped
class PushNotificationService(
    private val logger: Logger
) {
    @Fallback(fallbackMethod = "announceSecondaryPushNotificationService")
    fun announcePrimaryPushNotificationService(): Date {
        val fail = Random.nextBoolean()
        if (fail) {
            error("Can't reach primary push notification service")
        }
        logger.info { "Push notification estimated in 1 sec" }
        return Calendar.getInstance().apply {
            add(Calendar.SECOND, 1)
        }.time
    }

    @Retry(maxRetries = 2)
    fun announceSecondaryPushNotificationService(): Date {
        logger.info { "Push notification estimated in 3 sec" }
        return Calendar.getInstance().apply {
            add(Calendar.SECOND, 3)
        }.time
    }
}

Let's re-check the logs:

2022-07-24 16:03:53,936 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:53,994 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,000 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=1, last=1, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,001 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=1, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,460 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,463 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=2, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,464 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=2, last=2, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,579 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:54,937 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:56,180 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,185 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=3, last=3, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:56,186 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=3, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,618 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,622 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=4, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,625 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=4, last=4, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,105 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,110 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=5, last=5, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,111 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=5, deliveryTag=\x02, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:57,219 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:57,534 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,538 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=6, last=6, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,539 FINE  [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=6, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:58,039 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:58,041 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], Danger - Successfully verified a snake
2022-07-24 16:03:58,052 INFO  [de.fay.sna.PushNotificationService] (vert.x-eventloop-thread-0) Push notification estimated in 3 sec
2022-07-24 16:03:58,420 INFO  [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard

If it is "Danger - Successfully verified a snake" or "False positive - Successfully verified a lizard" inside the log: we always know if we verified a snake. The push notifications are estimated with 1 second for the faster and less reliable primary version and with 3 seconds in case the secondary fallback version succeeds within the two retries.

Adding resilience to Quarkus using Kotlin is quite easy, cause most of it can be done by using annotations and loose coupling.

If you want to try the resilience example from above, you can find it on Github. Have phun and stay curious.

Epilog
Snakes are cool animals. The snake detector is just a fun fiction and does not have the intent to support fears when it comes to wild animals. Next time I'll choose a scenario helping animals :) - cause we should protect all of them. Buy the way: Tardigrades and ants are good examples for very resilient animals!

Resilient microservices with Quarkus and Kotlin - Design Patterns

Felicia Faye — Tue, 05 Jul 2022 11:25:30 +0000

In the first part of the series, we took a closer look on why it is a good strategy to choose Quarkus in combination with Kotlin. Do you also remember the snake scenario? We wanted to create more resilient microservices then expected by default - just to make sure that we're save most of the time.

A good starting point for the resilience aspects are resilience design patterns. Most of them have the intention to reduce down times and speed up start up times, so that for example containers running in a cluster have a good uptime and can recover rapidly ⚡︎.

Resilience design patterns can be grouped into four main sections:

Isolation
Loose Coupling
Supervision
Latency Control

Isolation patterns are focused on stuff like parameter checking, bulkheads and shed loads. Loose Coupling stands for async communication, idempotency and an event-driven architecture. Supervision patterns deal with monitoring, escalation and error handling, while latency control is about retry, fallback and timeout mechanisms including patterns like circuit breakers and fan in/out strategies.

We are mainly going to have a closer look to the latency control patterns, but also check out some of the other sections, which help us to build services more resilient.

First, there is the retry pattern, where we would simply retry what failed before. In case of our snake scenario imagine a snake detector service, which needs to call a push notification service, which isn't very reliable.

As a solution we could just retry until we succeed.

This strategy is suitable for issues like:

temporary network issues and packet loss
internal server errors of the target system
no or slow response times of the target system

But wait .. what if the target system is already overloaded? Sometimes, it can be a bad idea to just increase the number of calls to ensure everything is working and even then, it is still not guaranteed.

So, let's better check another option: the fallback pattern, which uses another alternative option in case of a failure. In our case as we rely on an external push notification service.

We can just call an alternative service, which is maybe more expensive, but also more reliable in terms of service. Sometimes it also valid to have no fallback as fallback. E.g., when a shopping order service should do an external fraud check, but the fraud check is not that important, that the shop should reject orders for at least smaller amounts of money to ensure a great shopping experience. But in our case, we must ensure that we successfully call a push notification service.

Then, there is also the timeout pattern. This is quite well known from the HTTP world - especially used by HTTPS. It describes the process of waiting for a certain time to get a successful answer, but then give up actively. In scenarios where we cannot be sure that the call made it through or when we deal with slow servers it can be very helpful to configure and finetune the communication timeout values on a service level.

However, finding the right value can be difficult: we want the timeout to be long enough for slower responses, but giving up actively after a certain while. Tracking failure statistics can help to finetune the proper timeout values.

Circuit breakers are another pattern from the latency control family. They are inspired by the electrical breakers and do quite the same: they can sit between two components and control the throughput by being either open, half-open or closed.

The open state means that the circuit breaker itself is open and active and therefor all requests are blocked and cannot pass like in the illustration below.

A circuit breaker is half-open, when only a certain number of requests are allowed to pass.

When the circuit breaker is closed, all requests go through without any restrictions.

This concept can help to protect systems from shutting down due to overload. It is especially helpful in combination with the retry, fallback and timeout patterns.

Another very helpful pattern is called fan in / fan out and it is also a good example to demonstrate why we want loose coupling. Without loose coupling it would be complicated to apply fan in / fan out, because we need independent components for this: the pattern is about scaling workers or pods for producing and consuming independently. So, in this case fan in means that if producers are too slow and consumers are fast, the producers should be scaled up and maybe the consumers should also be slowed down. Fan out means that when producers are fast and consumers are slow, we want to scale up our consumers and additionally maybe have less producers. This way a system can handle different system loads for example at different day times by scaling workers up and down.

Now, after we've seen some helpful design patterns in theory, it's time to evaluate how we can realize them with Quarkus and Kotlin to build microservices more robust: we'll see that in part 3.

Resilient microservices with Quarkus and Kotlin - Introduction

Felicia Faye — Thu, 26 May 2022 10:39:00 +0000

Stable microservices are not always literally essential for survival, but imagine you live in an area where there are poisonous snakes, and you have many many detectors in your backyard warning you about them. The detectors are communicating via a Wifi connection and REST interface with your server - so this communication should be really robust.

How would you write the code for the server-side microservice?
Which technologies and languages would you use?

In this series I'll show how to implement resilient microservices using Kotlin and Quarkus. This is the first part of the series giving a short introduction to the technologies we are going to use and why they are a good choice implementing the stated scenario.

Why Kotlin?

First of all, let's see why we can benefit from using Kotlin as a primary language. Kotlin had its first official 1.0 release in 2016. Already then it had many improvements over the current state of Java at that time like:

Build-in null safety
Data classes
Native support for functional programming
Extension functions
Smart casts

just to name a few. Here is a small example of Kotlin in use:

data class User(
    // non null property
    val name: String, 
    // Int instead of int; no usage of primitives. 
    // But compiled to Javas int           
    val age: Int,  
    // nullable property              
    val description: String?     
)

val user = User(
    // named parameters
    name = "John Smith",  
    age = 30,
    description = null
)

// destructuring of data classes
val (name, age, description) = user    

val descriptionToPrint = description
        // null safe access to description with ?.
        ?.let { "the following description: $it" } 
        // fallback value for a null description 
        ?: "no description"    

// String builder
println("$name is $age old and $descriptionToPrint")    
// prints: John Smith is 30 old and no description

On top of that there are many more features available to play around with. Since its early releases the language kept on evolving and new features were added. Most importantly Coroutines and Flow as well as more recent entries into the language like

Value classes
Contracts

or one of the newest additions Context Receivers, still in an experimental development phase. On top of that it fits quite nicely into my personal ecosystem, which is driven by the facts that:

I'm mainly a mobile developer
I have a strong interest on cross-platform solutions
I like the Android platform the most
Most apps work better with a dedicated backend for frontend, so I write those too.

Kotlin provides everything needed for this.

In the native world, Kotlin gained a lot of tracking with the announcement of Google to make it a first class citizen for Android development and later on the leading language, mostly replacing Java in that area.

More recently Kotlin also introduced a solution for cross-platform development in Kotlin Multiplatform, which enabled Kotlin to run the the same code across Android, iOS, Web and Desktop.

Since Java is still strong in the area of backend development many of the frameworks used there are either able to be used with Kotlin or have additional support for Kotlin's unique features. Such frameworks for example are Quarkus, Spring Boot or Micronaut. Also already matured frameworks that are native to Kotlin, like Ktor, are available for developers to choose from.

Why Quarkus?

Let's have a closer look on Quarkus and why it can help us to fulfil our requirements. What is Quarkus? It provides an easy approach to create a scalable backend on an enterprise level.

Generally speaking it's a Cloud Native Java stack, which is built with many mature and well known (open source) Java libraries, frameworks and standards. So instead of learning everything new, we can use for example Vert.x, Camel, Hibernate and RESTEasy. In case of standards we can find MicroProfile and selected specifications from Jakarta EE. Dependency injection uses the well known CDI. Looking at annotations, we can use JAX-RS annotations when defining REST endpoints. On one hand JPA annotations can be used to map persistent entities. On the other hand JTA annotations are there to declare transaction boundaries with Hibernate. Last but not least there is SmallRye to configure, secure and monitor our application.

They say it's

supersonic subatomic Java

Puh Java what?!

Instead of Java only, it also works well with GraalVM and Kotlin YAY!

We will have a closer look on how to do this, later.

What's the idea behind Quarkus?

The idea behind Quarkus is to optimize Java for cloud native environments.

This is done by several optimizations during build time, a failed startup intention and the fact that it has Vert.x underneath.

Sounds quite good when running apps in containers!

What does Quarkus to optimize the build time? During build time Quarkus removes everything which is not needed. Additionally dependency injections are analysed and already injected at compile time - so we could say it is hard wired.

With their fast startup intention, Quarkus realizes very fast start up times, which is ideal for cloud native environments: when scaling up for launching more containers or in a Recovery scenario for restarting containers.

Additionally because Quarkus has Vert.x underneath, it relies on the reactor pattern, so it can handle a lot of requests with ease. Good for us when a lot of requests are coming in.

From my perspective

Quarkus offers hot reload, which is not self evident in the Java world, but feels like state of the art nowadays. I usually use it on a daily bases, so it is pretty important to me.

Quarkus has an active and supportive community, where we can contribute and exchange information rapidly. Having a feature request yourself to e.g. enhance Kotlin support? Just talk to them and in the best case offer a PR yourself. You will surely be heard.

With a long history of writing Java backends, Quarkus makes me also feel at home with their alignment to the JEE specs (Annotations, Extensions, Frameworks, Libraries ...) and especially with RESTeasy. This even does not cause any performance disadvantages, as everything is mapped to Quarkus cloud native at build time. But in contrast to JEE approach to handle annotation processing during runtime, Quarkus is performing these operations during build time which drastically improves the performance of the quarkus app.
If that is not enough, you are still able to compile Quarkus to a native binary which improves the performance even more.

Imperative and reactive

Quarkus gives us the freedom to implement our application in an imperative style, which is well known from JEE, but it also allows to use a more reactive approach - leveraging the power of Kotlin Flows.

Imperative (REST endpoint)

@Path("greeting")
class GreetingEndpoint(
    private val greetingService: GreetingService
) {
    @GET
    @Produces(MediaType.TEXT_PLAIN)
    fun greeting(@QueryParam("username") username: String?) = 
            greetingService.greet(username)
}

Reactive (Server sent events)

@GET
@Produces(MediaType.SERVER_SENT_EVENTS)
@RestSseElementType(MediaType.TEXT_PLAIN)
fun stream(): Flow<String> = flow {
        emitAll(channel.asFlow())
    }

Easy project setup through code generation

Getting started is quite easy by visiting https://code.quarkus.io/. As a first step start searching for Kotlin and select the related checkbox.

After that go searching for RESTeasy-reactive and make sure that the related checkbox is also selected.

Nice! Our app is ready for download.

Now we can start writing our first endpoint.

@Path("greeting")
class GreetingEndpoint(
    private val greetingService: GreetingService
) {
    @GET
    @Produces(MediaType.TEXT_PLAIN)
    fun greeting(@QueryParam("username") username: String?) = 
            greetingService.greet(username)
}

The code sample above gives us a GET endpoint with a query parameter for the username and produces a greeting response message with media type TEXT_PLAIN.

To make it run just execute

quarkus dev

in your command line. Use a rest client like postman to test the response.

But how to make this resilient?

We'll see that in Part 2 by highlighting the resilience design patterns that do the most to reduce recovery times and down times - stay tuned.

In the meantime, if you want to try my welcome example from above, you can find it on Github.