DEV Community: Sergio Garcia Moratilla

Modernizing a CI+CD pipeline with Github Actions

Sergio Garcia Moratilla — Tue, 07 Jun 2022 14:41:33 +0000

Our CI+CD has been working for 5 years long. You know, if it ain't broken, don't fix it. But the company is not the same. It's time to update it!

Let me wrap up a bit our current setup. I am going to be brief, I promise. We have a Jenkins cluster on-premise. That is, we manage (and maintain) a bunch of hosts that run Jenkins slaves and a host that runs the master. They are within our on-premise VPN, a reminiscence of our first hosting provider.

Our production backend runs a containerized system on top of a Docker Swarm cluster. Our container registry is Nexus, which allows us to deploy our services in the Swarm.

Both systems run within their own independent VPNs.

Our code repositories are in Github. Our Jenkins is listening for changes in Github. When we merge to develop/production, Jenkins pushes the image to the Nexus, connects via ssh to the managers of the cluster and runs the deployment command (docker stack deploy). We have a common Jenkins pipeline for all our services.

Pros:

Simple.
Security on our side (VPN certificates + ssh keys are in our servers).
Jenkins is commonly known.
Stable: we haven't had to change it a lot all this time.

Cons:

We need to maintain and keep update the Jenkins machines.
As the team grows, the Jenkins cluster has to grow too to be able to run more jobs.
We have only configured Java 8 and 11.
Still on our old hosting provider.

We have been pretty happy with this setup so far. Maintenance is something that we always want to simplify at Playtomic. The fewer systems, the better. Besides, new versions of Java require new versions of the JDK, maven and thus... Jenkins. We have already been using Github Actions in other projects here, so that we know that it could be a fine replacement of Jenkins. The workflow defines the environment it is required to run (for example, the java version or the architecture), so it makes complete sense to use Github Actions.

Replacing Jenkins by Github Actions

We re-wrote the Jenkins pipeline as a Github Actions workflow. Our main concerns are two:

What about security? We are using ssh-keys to access the cluster. We can use Github secrets to store them, but we don't like having such an important piece of the security stored in a third-party.
Cost might be a problem in the future. Github Actions are charged by the minute of computation and we have 50 services.

We discovered that Github Actions allow you to run self-hosted nodes... so that's the solution to both problems. At this very moment, we can afford the minutes that we are spending, so we are adding a host just for deployments. We added a t3.nano, which is pretty cheap. ssh-keys are still 100% under our control, as they are installed in the machine.

Pros:

Still a common pipeline (workflow) but allows everyone to use new stuff (new java version, different languages, ...) without installing more tools in Jenkins.
Security is still on our side.

Cons:

We need to monitor the cost.

What's next?

Containerized Github Action runner

Worried about the cost? If you have a cluster, you can run as several copies as you want of the Github Action runner!

On-demand self-hosted runners

This would be a huge improvement to control the cost while still being able to scale the number of runners:
https://github.com/machulav/ec2-github-runner

Separate the CI from the CD

To separate pipeline for the CI and CD is a good idea, as their lifecycles are pretty different.
In our current setup, the workflow is responsible for running the deployment command. If that fails, then the whole pipeline fails, but the build was successful.

We have already tested ArgoCD in Kubernetes so that the CD is handled by the cluster itself.
If you are running Kubernetes, you can already do that with FluxCD or ArgCD

Replace Nexus by Github Packages

Nexus requires a lot of space (as it stores all the containers, libraries, packages, ... of your organization). We don't want to maintain that space, be responsible of the backups, ... so that we are considering to migrate to Github Packages too.

Let's talk about performance and MongoDB

Sergio Garcia Moratilla — Wed, 16 Feb 2022 10:21:38 +0000

If you, a backend developer, had to describe your job, what would you say? We usually talk a lot about servers, clusters, layers, algorithms, software stacks, memory consumption, ...

We put data into databases and we get it back as fast as we can.
Databases are our cornerstone. Why don't we talk more often about them? We are always relying on our ORMs.

My best advice? Simplify your queries. Simplify your data models. Simplify your access patterns.

Performance in MongoDB

Metrics

If you are using MongoDB, these two metrics will be your best friends:

Scanned Documents/Returned ratio
IOPS

Scanned docs/returned: it means how many documents you are reading from disk vs the number of documents you are actually returning in your find() or aggregate(). Ideally: this should be 1 (every document read is returned). The only way to get it? All your queries must be covered by indexes. Indexes are in memory (or they are if they fit), so MongoDB doesn´t have to read and filter them from disk.

IOPS: I/O operations per second. That is, operations on disk. It is correlated with scanned docs, as they are read from disk. But there are more sources of IOPS, for example writes.
Your disk will give you a limit to your maximum IOPS. Ours is 3000.

Your goal is to keep IOPS below that threshold, and as low as possible. It is hard to know how many IOPS your query consumes, but it is easy to know the scanned docs/returned ratio.

Tools

How can you analyze why your database is behaving as it is?

MongoDB Profiler
If you can afford to enable it, do it now. It's the best source of info. We use the Atlas MongoDB Profiler and it is worth every penny.

Explain planner
The planner is the key to understanding your access patterns.
There are several ways of calling it, but you can start with explain() after your cursor:

db.our_collection.find(query).explain()

You can use add executionStats to get more data about the query.

db.our_collection.find(query).explain("executionStats")

There are many stages, but these ones are the most important:

COLLSCAN: the query is scanning the collection in disk. Pretty bad, as no index covered the search, so MongoDB has to read the whole collection.
IXSCAN: the query is using an index to filter. It doesn´t mean that all the query is covered by the index, but at least some part.
FETCH: the planner is reading the documents from the collection. If your query is returning documents, you will get a FETCH stage probably (unless your query is covered by the index).

This is an example of one of our queries:

winningPlan: 
      { stage: 'COLLSCAN',
        filter: 
         { '$and': 
            [ { '$or': 
                 [ { 'invitaed_user_id': { '$eq': '1' } },
                   { owner_id: { '$eq': '1' } },
                   { 'player_id': { '$eq': '1' } } ] },
              { is_canceled: { '$eq': false } },
             ] } },

This is not good, as it is scanning the whole collection.

Another one:

db.our_collection.find(query).explain("executionStats")
executionStats: 
   { executionSuccess: true,
     nReturned: 22,
     executionTimeMillis: 0,
     totalKeysExamined: 24,
     totalDocsExamined: 22,
     executionStages: 
      { stage: 'FETCH',
        nReturned: 22,
        docsExamined: 22,
        inputStage: 
         { stage: 'OR',
           nReturned: 22,
           inputStages: 
            [ { stage: 'IXSCAN',
                nReturned: 0,
                indexName: 'example-index-1',
                indexBounds: 
                 { owner_id: [ '["1", "1"]' ],
                   start_date: [ '[MaxKey, MinKey]' ] } },
              { stage: 'IXSCAN',
                nReturned: 22,
                indexName: 'example-index-2'
                indexBounds: 
                 { 'player_id': [ '["1", "1"]' ],
                   start_date: [ '[MaxKey, MinKey]' ] ] },
                keysExamined: 23,
                dupsDropped: 0 }] } } },

Here you can two see IXSCANs, merged by and OR. After that, the query is fetching the documents. I am reading the query inside-out. example-index-1 is used to resolve one part of the query, and example-index-2 for the other part.

Sometimes you will get a FETCH just after an IXSCAN: it means that the index covers only part of the filter. After that, the planner needs to read the documents from disk to finish the filter.

The complete list of stages? You need to check the code.

Simplify your queries

$ors

$ors are the devil. You, as a programmer, are used to thinking in $ors. You add a few $ors, your condition gets much more expressive. But guess what? Your query has gotten exponentially more complex. With every condition you add to the $or, you are adding one more combination of parameters.

How does the planner resolve all those combinations? It needs an index for each of them.

Do you remember the explain() above? The query was:

{
    $or: [{"owner_id: "1"}, {"player_id: "1"}].
}

The indexes used?

example-index-1: {"owner_id: "1", "start_date": 1}
example-index-2: {"player_id: "1", "start_date": 1}

And here is another tip: bigger indexes can cover smaller queries as long as the fields are at the beginning of the index.
We are not using start_date to filter.

What would happen if I add an extra $or?

{
    $or: [{"owner_id: "1"}, {"player_id: "1"}].
    $or: [{"is_canceled": true}, "start_date": {$gt: ISODate("2022-02-02T00:00:00)}]
}

Then there would need 4 combinations:

{owner_id, is_canceled}, 
{owner_id, start_date}, 
{player_id, is_canceled], 
{player_id, start_date},

Denormalized fields

If you find yourself filtering by several fields within an $or, or sorting by several fields, consider adding a denormalized field (based on the others).

Yeah, my apologies to the third normal form.

Keep reading, we will give an example pretty soon using the so-called Summary Pattern.

nulls are always the minimum value

This is a minor trick, but still useful. Let's say you have a nullable field, and you have to filter (or sort descending) by that field.

You will probably find yourself using a query similar to this:

{
 $or: [
        field: {$exists: false}, 
        field: {$lte: value}
      ]
}

That is, if the field is null, then it passes the filter. Otherwise, compare. We used this filter to check if a player could join a match given their level and the level restrictions of the match.

That's an $or. We don't like $ors. What would happen if we compare our value to null? Let's check the sorting rules in MongoDB.

MinKey (internal type)
Null
Numbers (ints, longs, doubles, decimals)
Symbol, String
Object
...
MaxKey (internal type)

That is, null is always the lower value when comparing (except for the MinKey object, we will talk about it later).

Our query can be simplified by this:

{
  field: {$lte: value}
}

It works for $lt and $lte (lower than and lower or equal).
It works too if you are sorting by descending order of field {field: -1} because the object with null will be at the end of the sort.

Counts are costly in MongoDB

Counting seems an easy operation, but it is not. Even if you have an index, MongoDB needs to traverse the index due to the way MongoDB builds B-trees: they don't store the number of leaves that the sub-tree have. So they need to traverse the index until the end.

Again, if you are counting using a query with $or, it makes the counting even more complex: the query needs to take into account possible repeated documents.

For example, we used counts to compute the position of a player in a ranking (the original query was even more complex).

{ranking_id: ?0}, 
{
 $or: [
        { value: {$gt: ?1}},
    { value: {$eq: ?1}, last_modified: {$gt: ?2} }
 ]
}

It required several indexes to count:

{ranking_id: 1, value: -1}
{ranking_id: 1, value: -1, last_modified: -1}

How can we avoid counting on several indexes? Add a single field that summarize the fields that you are filtering.

For example: weight = append(value, last_modified)
With that new field, we only required one single index:
Indexes:

{ranking_id: 1, weight: -1}

This is called the Summary Pattern.

How to build indexes

Ok, indexes are our best tool to keep MongoDB as performant as possible. So the next step, how do we know what indexes we should build?

Performance advisor

If you are in Atlas, use the Performance advisor.

At some point, you will know your system better than the Performance Advisor, but it is a good starting point.

Clone your production collections and explain() it in local

Test your indexes thoroughly before you put them in production: - it takes a lot of IOPS to build them.

once you built it, the planner is taking it into account too, even if it is not finally used.

Remember what we said before:

COLLSCAN: bad.
IXSCAN: good.
FETCH: good if it is the final step. Bad in between. Others that you will see:
COUNT: ok.
MERGE: ok-ish, you probably could do better.
MERGE_COUNT: good.

By the way, there are blocking and non-blocking stages, meaning that a stage needs to wait for the previous one before it can start computing results.

ESR: Equal-Sort-Range

Have you ever wondered what fields should go first in an index? You need to follow this rule:

Fields filtered by equals ($eq, $in in some cases, ...) go first.
Then fields used in the sort stage. Remember that the index has to be built in the same order as you are sorting.
Then fields filtered by a range ($lt, $lte, $gt, ...).

ESR is the most useful rule you will find to build indexes. You should read as much as you can about it until you understand it.

This post by Alex Belilacqua is a gem.

Disambiguate equals

If you have two fields that are going to be filtered by $eq, what should go first?

The answer is that it doesn't matter. You don't need to worry about having a more balanced tree.

Just keep in mind the ESR rule. If one of them goes in a sort or a range, then it goes the latter.

Rollover process

Building an index is one of the most costly operations. Your IOPS will go nuts. If you need to do that in your production environment, and your collection is big enough, then we recommend you to use a rolling process. It starts the index build in a secondary, then promotes it to primary once the build is finished. You will be able to build any index even when your database load is high.

In Atlas, it's just one click.

Remove / Hide indexes

The more indexes you have, the worst for the planner. The planner runs the query through the indexes it has and then takes the most promising.

Again, counting when you have several indexes is pretty bad.

Sometimes, you cannot just remove an index in production. You can check using your profiler if it can be removed but you might not be 100%. One not-frequent query might launch a COLLSCAN and then you would miss that index.

Luckily, since MongoDB 4.4 you can hide indexes. We use them to detect what indexes we can remove safely.

Limit your queries

Have you set a maximum limit of time to your queries? Why not? Do your clients have a request time out? Then it can be a healthy practice to avoid unexpected uses of your APIs.

db.our_collection.countDocuments(query, {maxTimeMS: 100})
MongoServerError: Error in $cursor stage :: caused by :: operation exceeded time limit

Do you see all these orange dots? No one was waiting for the backend to reply.

References

Flux vs ArgoCD

Sergio Garcia Moratilla — Thu, 07 Oct 2021 08:01:03 +0000

Hey there!

Do you have any war stories about Flux or ArgoCD? Anytime they failed miserably? Did you have to fix manually your cluster? We want to hear those stories!

Why do we ask?

We are exploring this GitOps trend at Playtomic. We have two k8s clusters running using Flux and ArgoCD. We have a multi-tenancy setup, that is, one repo for the cluster itself, one repo for every application we deploy in the cluster. Flux and ArgoCD image updaters are in place.

To be honest, both are pretty similar and any of them would work for us. If you search for comparisons between Flux and ArgoCD, you will only get a few of boring list of features. So... we are looking for first hand experiences.

Why did you choose one over the other? Did you have the chance to try the other one?

Hip hop to code

Sergio Garcia Moratilla — Tue, 14 Sep 2021 20:35:21 +0000

I have read tons of posts about music to code.

Some people find focus in the sound of the rain. Some prefer electronic music (techno, house, ...). Some use lo-fi (I am thinking of you @angelolloqui). I have a few crazy friends listening constantly to heavy metal (maybe they want to destroy everything after all).

I have never heard of anyone proposing my new favorite genre: hip hop instrumental music. Specifically freestyle beats.

If you are going to give just one shot to this kind of music, check this one.

If it doesn't make you move your neck, you're already dead inside 🤪

Jokes aside, what kind of music do you listen to code? Do you have something new that can surprises us? Would you like to share some beats with me? 😇

Header: Kaysha

The path to observability

Sergio Garcia Moratilla — Mon, 02 Aug 2021 09:21:01 +0000

o11y = observability = logs, metrics, traces

This post is a summary of the steps we unconsciously followed in Playtomic when digging into the world of SRE (Site Reliability Engineering).

Logs

Most of the people is already comfortable with logs.
I have always been a fan of the 12-factors app design, and they have a chapter about logs. Just throw them to the stdout
and treat them as a stream. As our services run in docker containers, we just read the logs from docker. With logstash+filebeat you can massage and forward them somewhere else (an elasticsearch, logz, datadog, ...)

anemone_configuration.1.ip.eu-central-1.compute.internal    | 2021-05-25 08:44:44.080  INFO [configuration-service,65360f43df5797ee,65360f43df5797ee,3187176603182004136,7845016010104016623,767246] 1 --- [  XNIO-1 task-5] c.p.a.s.IgnorableRequestLoggingFilter    : After request [GET uri=/v2/status/version_control?app_name=playtomic&app_version=3.13.0&device_model=iPhone&os_version=14.5.1&platform=ios;user=one.user@gmail.com;agent=iOS 14.5.1;ms=1]
anemone_configuration.1.ip.eu-central-1.compute.internal    | 2021-05-25 08:44:44.087  INFO [configuration-service,5fe21e342fb8cac9,5fe21e342fb8cac9,4868272858418826990,131548911514375389,767246] 1 --- [  XNIO-1 task-6] c.p.a.s.IgnorableRequestLoggingFilter    : After request [GET uri=/v2/configuration;user=one.user@gmail.com;agent=iOS 14.5.1;ms=8]

Metrics

Metrics are also quite known in the community, thanks to prometheus and grafana. What is most unknown and super useful is to have them correlated somehow.
Probably via tags/attributes to identify hosts, containers, environments, ...

Traces

Traces are probably the newest in the trio.

In Playtomic, we followed the standard path: we started with in-house ELK (for logs) and prometheus (+ grafana). But we (developers) hate maintaining the infra: we run out of space, indexes get corrupted, ... it is quite distracting. Therefore, we got rid of these in-house versions and moved those services to Datadog.

At that moment, we were already using distributed tracing to inject the trace and spain ids in our logs, so we can keep track of the requests among our microservices. So, with the Tracer already in place and the use of Datadog, the next natural step was using Trace to know more about our platform.

What's next?

This SRE world is huge. Where are we heading to?

SLIs, SLOs, SLAs

Once you have your data (specially metrics), you can monitor your system. Welcome to the world of SLIs, SLOs, and SLAs.

SLI: Indicator (Service Level): the number (for the tech team). Example: error_rate = % of requests with code >= 500.
SLO: Objective (Service Level): the target (for the tech team). Example: keep error_rate < 99%
SLA: Agreement (Service Level): the agreement (with the customer). Example: keep error_rate < 95%

SLOs are great because it gives you a hint when to spend more time in stability (paying technical debt, investing time researching problems, ...).

RUM: Real User Monitoring

One step further: what if you could connect traces in your clients with your traces in your server? That's where we are at this moment.

Liquibase: Don't use ISO dates as changeset ids with yml

Sergio Garcia Moratilla — Tue, 21 Jan 2020 10:09:01 +0000

After upgrading a service from java 8 to java 11, our liquibase got crazy and tried to apply every change to the database from the beginning. But they were already applied and no changes were made to the database.
We discovered that using ISO dates as identifiers of the changeset was a bad idea. The hard way.

Liquibase Changesets

Changesets in liquibase are, as their name implies, a set of changes that must be applied at once in a database (or not at all).

In order that liquibase is able to distinguish when a changeset is already applied or not, it requires you to set a unique identifier.

By convention, we started to use the ISO date when the change was written (format yyyy-mm-dd).

For example:

databaseChangeLog:
- changeSet:
    id: 2019-06-06
    author: sergiomoratilla
    comments: Adding new value to know which client was used to book.
    changes:
    - modifyDataType:
        tableName: reservation
        columnName: client
        newDataType: ENUM('WEB_DESKTOP', 'WEB_MOBILE', 'APP_ANDROID', 'APP_IOS', 'UNKNOWN') NOT NULL

Liquibase stores the applied changeset in a table called DATABASECHANGELOG within your schema. So that, we were expecting a row

"2019-06-06"    sergiomoratilla /liquibase/changelogs/this-example.yml ...

It's a trap

Liquibase's YML parser decides to get the changeset id as a Date, and then use toString() to generate the id. That smells because you don't have control on that format... and this is exactly what happened.

Instead of storing "2019-06-06" we got "Wed Jun 06 00:00:00 GMT 2019".

After upgrading to Java 11, the behaviour of toString() changed and now returns "Wed Jun 06 02:00:00 CEST 2019". It is exactly the same date so that this format is correct but it is a bit weak to trust your ids to default formatting.

Solutions

Don't use ISO date as ids (if you are using yml format to configure that). Probably, most of you didn't already do that.

When we started to have several changes on the same date, we decided to change that format to yyyymmdd-n, where n is an incremental integer.

What if you are already using them? I suggest you to replace the ids in your changelogs files by the ids you already have in your database. And change your convention for new files!

Weather data providers: which one?

Sergio Garcia Moratilla — Fri, 27 Dec 2019 11:46:38 +0000

We are planning to provide our users with weather forecast before they book at Playtomic.

To do that, we need a weather data provider O:) We have looked into these 3 (probably the most known):

Pricing is not an issue right now. We intent to cache our queries. What I'm worried about is data accuracy.

Any experience that you would like to share? It would be really appreciated!

A way of testing code based on time in a Spring Boot Application

Sergio Garcia Moratilla — Fri, 20 Dec 2019 12:42:48 +0000

Every time I see a Instant.now() / new Date() / something that creates a date based on the current time I tremble. How do you expect to test that in a simple / coherent / easy to follow way?

When I see that code, I usually see tests using now() + duration to check that everything works. So that the test is not the same when run today than when run tomorrow.

Wouldn't it better to be able to "fix" the time and test your code with exact values of times/periods as well?

So that I've fighting in my team in order that everyone uses a ClockProvider to get the "current" time. That way I can instance my own ClockProvider in unit tests and I can override the default ClockProvider of my application in integration tests.

The former is pretty easy to understand, so that I'm not writing an example of that. This is an example of the latter.

For instance, my application would look like this:

@SpringBootApplication
public class BookingApplication extends AbstractAnemoneApplication {

    @Bean
    @ConditionalOnMissingBean(ClockProvider.class) // to allow tests to overwrite it
    public ClockProvider getClockProvider() {
        return () -> Clock.systemDefaultZone();
    }

    // An example of method using ClockProvider
    boolean isItJanuary(@Nonnull ClockProvider clockProvider) {
        return clockProvider.getClock().instant().atZone(UTC).getMonth() == Month.JANUARY;
    }

And my IT tests:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, classes = {
    FixingClockConfiguration.class, BookingsApplication.class})
@ExtendWith(SpringExtension.class)
public class BookingsApplicationIT {

    @TestConfiguration
    static class FixingClockConfiguration {

        @Bean
        public ClockProvider getClockProvider() {
            ZonedDateTime fixedInstant = ZonedDateTime.of(2019, 04, 15, 14, 00, 00, 00, UTC);
            return () -> Clock.fixed(fixedInstant.toInstant(), fixedInstant.getZone());
        }
    }

// ... your tests based on that date
}

Image: https://unsplash.com/photos/FfbVFLAVscw

Concurrency in Spring's StreamListener and Kafka

Sergio Garcia Moratilla — Tue, 17 Dec 2019 19:08:20 +0000

TL;DR: go to Use Configuration

Another too fast, too furious post. I have spent a few hours trying to make my event processor multi-threaded, and it's so damn easy that I don't want anyone to spend more than a few minutes.

We are using the Spring Cloud Stream layer to configure our Kafka consumers.

For example, a configuration for a processor named 'reservations-input' connected to a Kafka topic 'reservations-topic' would be similar to this:

spring.cloud.stream:
  bindings:
    reservations-input:
      content-type: application/json
      destination: reservations-topic
      group: consumer-service-group

And your class to start processing those events:

@EnableBinding({
    MessagingConfiguration.ReservationTopic.class})
public class MessagingConfiguration {
    public interface ReservationTopic {

        String INPUT = "reservations-channel";

        @Input(INPUT)
        SubscribableChannel input();
    }
}

@Service
public class ReservationProcessor {
    @StreamListener(MessagingConfiguration.ReservationTopic.INPUT)
    public void handle(@Nonnull Message<ReservationEvent> reservationMessage) {
        // your stuff
    }

Easy peasy. Only problem here is concurrency.

If you have used Kafka before, you would know that the number of partitions in your topic limits the concurrency.
Each partition have 1 single consumer.

I don't know whether (or where) I read that, but I assumed that my application would generate as many threads/consumers as partitions my topic has. But I was wrong. By default, Spring's only generates 1-threaded processor.

Solutions? Get more instances of your application or configure ConcurrentKafkaListenerContainerFactory to be able to throw more threads (see https://docs.spring.io/spring-kafka/docs/2.3.x/reference/html/#container-factory).

Option 1: create your own instance of ConcurrentKafkaListenerContainerFactory.

The only hint I found in the documentation or stackoverflow but to instance a bean of type ConcurrentKafkaListenerContainerFactory

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Object> kafkaListenerContainerFactory(
        @Nonnull ConsumerFactory<String, Object> consumerFactory) {

        ConcurrentKafkaListenerContainerFactory<String, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        factory.setConcurrency(5);

        return factory;
    }

I am not very prone to instance my own beans to configure things that seems too obvious. It is easy to overwrite some Spring default values that I am already expecting to use, it is more code to maintain...

There has to be a way through configuration.

Option 2: use configuration

Getting back to configuration, what we write under spring.cloud.stream.bindings.channel-name.consumer ends in the configuration of Kafka. So that I tried to configure the property concurrency. That is:

spring.cloud.stream:
  bindings:
    reservations-input:
      content-type: application/json
      consumer.concurrency: 3
      destination: reservations-topic
      group: consumer-service-group

Starting our application, we see that we have 3 binders.
Profit!

    December 17th 2019, 14:22:57.274    2019-12-17 13:22:57.274  INFO [consumer-service,,,] 1 --- [container-1-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : partitions assigned: [reservations-topic-1]
    December 17th 2019, 14:22:57.259    2019-12-17 13:22:57.259  INFO [consumer-service,,,] 1 --- [container-2-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : partitions assigned: [reservations-topic-2]
    December 17th 2019, 14:22:57.256    2019-12-17 13:22:57.256  INFO [consumer-service,,,] 1 --- [container-3-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : partitions assigned: [reservations-topic-3]

BlockingQueue and ExecutorService

Sergio Garcia Moratilla — Mon, 16 Dec 2019 13:27:23 +0000

This is a quick and dirty post, but I have promised to publish everything I research at Playtomic.

We were having a discussion about how to limit how many tasks an ExecutorService can enqueue. We were trying to
control how much memory a service can handle to avoid out of memory exceptions. This service accepts messages from a Kafka topic
and from an API. Those operations end in the same internal logic, which is threaded.

There is a kind of Queue, BlockingQueue, that can wait until a spot in the queue is free. It would seem that using an ExecutionService
with a BlockingQueue would wait when submitting a task until that queue is not full. But it is not, the ExecutionService rejects the task.

You know that hours of trial and error can save you hours of reading the manual. I'm proud to say that I have read the manual first this time.

This test shows what happens:

public class BlockingQueueExecutorServiceTest {

    @Test
    public void submitTest() {
        // Worst case scenario: accept only 1 thread in the queue.
        int nThreads = 1;


        ExecutorService exService = new ThreadPoolExecutor(
            nThreads,
            nThreads,
            0L,
            TimeUnit.MILLISECONDS,
            new LinkedBlockingQueue<>(nThreads));


        // Full this with tasks
        for (int i = 0; i < 10000; ++i) {
            WaitingTask t = new WaitingTask(i);
            exService.submit(t);

        }
    }

    private static class WaitingTask implements Runnable {

        int index;
        public WaitingTask(int index) {
            this.index = index;
        }

        @Override
        public void run() {

            try {
                log.info("Running task {}", index);
                Thread.sleep(1000);
            } catch (InterruptedException e) {

            }
        }
    }
}

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@550dbc7a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@4dbb42b7[Wrapped task = com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest$WaitingTask@66f57048]] rejected from java.util.concurrent.ThreadPoolExecutor@21282ed8[Running, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 0]

    at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
    at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
    at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
    at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
    at com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest.submitTest(BlockingQueueExecutorServiceTest.java:28)
... more boring stacktrace

If you want to wait until queue is not full, you have to provide a RejectedExecutionHandler which does that. For example, Spring's CallerBlocksPolicy.

public class BlockingQueueExecutorServiceTest {

    @Test
    public void submitTest() {
        // Worst case scenario: accept only 1 thread in the queue.
        int nThreads = 1;

        CallerBlocksPolicy policy = new CallerBlocksPolicy(10000); // 10secs
        ExecutorService exService = new ThreadPoolExecutor(
            nThreads,
            nThreads,
            0L,
            TimeUnit.MILLISECONDS,
            new LinkedBlockingQueue<>(nThreads), 
            policy);


        // Full this with tasks
        for (int i = 0; i < 10000; ++i) {
            WaitingTask t = new WaitingTask(i);
            exService.submit(t);

        }
    }

    private static class WaitingTask implements Runnable {

        int index;
        public WaitingTask(int index) {
            this.index = index;
        }

        @Override
        public void run() {

            try {
                log.info("Running task {}", index);
                Thread.sleep(1000);
            } catch (InterruptedException e) {

            }
        }
    }
}

And this time we get:

12:21:32.409 [pool-1-thread-1] INFO com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest - Running task 0
12:21:32.422 [main] DEBUG org.springframework.integration.util.CallerBlocksPolicy - Attempting to queue task execution for 10000 milliseconds
12:21:33.420 [pool-1-thread-1] INFO com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest - Running task 1
12:21:33.420 [main] DEBUG org.springframework.integration.util.CallerBlocksPolicy - Task execution queued
12:21:33.421 [main] DEBUG org.springframework.integration.util.CallerBlocksPolicy - Attempting to queue task execution for 10000 milliseconds
12:21:34.423 [pool-1-thread-1] INFO com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest - Running task 2

Header: https://unsplash.com/photos/Kj2SaNHG-hg

Choosing a graph database

Sergio Garcia Moratilla — Tue, 03 Dec 2019 18:39:36 +0000

Disclaimer: I am writing this because I throw a question about what graph database we should use at Playtomic to dev.to, and I would like to give something back to the community. This is the process of how we decided.

When starting with a new piece of tech, the question is always clear: which one of the alternatives? Most of you have had to choose a relational database: mysql vs postgresql vs oracle... Maybe, if you were lucky enough, a no relational database: mongodb (document) vs redis (key-value) vs cassanda (columns) vs hbase (columns)...

What about graphs databases (which are no relational as well)? I only knew Neo4j. With every cloud-vendor offering their own proprietary solutions (i.e. Amazon Nepture) this choose is even harder.

We prefer open source solutions over proprietary ones. More community, less prone to be vendor locked-in. We tend to host on cloud services. We don't have a infrastructure team and we don't want to spend time in maintenance.

In our philosophy, experiments must be goal oriented, not tech-oriented. What's the purpose of testing a graph database in Playtomic? We want to explore whether we can model relations between players better than we already do (with a relational database). Final aim is ending with a recommendation system: new players to meet, new venues where to play, ... all based in the relations with players that you already know.

As our team is small, our time to spend in experiments is limited too. So that, I have to reduce the number of options: Nepture is proprietary and pretty unknown to me, so that I will drop it. I'm not very seduced by OrientDB, as it looks like a too general db.

OrientDB:

Schema-less
SQL for queries (big win IMHO).
Great web console.

Janusgraph:

Schema.
Gremlin for queries (functional programming language)
Drawback: you have to choose what storage to use: HBase vs Cassandra. I'm not sure about the implications.
No web console. You have to use third-party application to visualise data.

Neo4j:

Schema-less (I'd rather say, only supports key-value).
Cipher for queries (query language).
Web console.
Hosted by GraphStory and GrapheneDb (cheaper).

All of them are available in AWS marketplace. Neo4j has an online sandbox which the rest don't have. You can play with all using docker images.

At this point, I think all three would give us all we need but Neo4j seems easier to test.
Searching how to integrate them with Micronaut and Spring Boot, Neo4j is more popular. It doesn't mean it is better, just that when I find a problem (and I will), someone has already been there.

Thinking about putting this experiment in production, we found some managed Neo4j hostings that allows us to start cheap and scale later if we are happy with it.

So that, Neo4j was the chosen one for our experiment. We will probably writing about the result in a few weeks x)

Graph databases: which one?

Sergio Garcia Moratilla — Tue, 15 Oct 2019 10:44:32 +0000

I have been considering graph databases for a while. I have minor experience with Neo4j, but I am a total newbie about what alternatives are in the market.

I see Amazon has Neptune, there exists OrientDB, MongoDB has incorporated graph functions, ...

Has anyone built a product on top of a graph database and would like to share how the experience was? Should I stick to Neo4j?