DEV Community: appaloosa.io

Active Record, Sidekiq, pools and threads

appaloosa.io — Thu, 07 Feb 2019 12:54:01 +0000

At Appaloosa, we use Sidekiq a lot. In fact, we use it for all the background jobs we need. We also use a database (yes, this sounds crazy, I know.) with Active Record as ORM. Sometimes, Sidekiq jobs need to do stuff in our database, this looks like a normal situation, until…

The recurring error…

Until, this error pops many times:

could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use (ActiveRecord::ConnectionTimeoutError)

And it pops many times, every day, at the same hour. Something like late at night, so yes, let’s try to understand and fix it.

But how can this happen?

Ok so, after checking all our monitoring tools we figured out that the error pops inside a Sidekiq job. And then, with more than 20 tabs opened on the browser (obviously, many were the same, opened from different sources, at different time), we found the theory and theoretically, the solution.

Theory

First of all, we have Active Record on one side with its proper configuration file. In this config/database.yml file, you can find something like:

Remember the error? could not obtain a connection from the pool. It must have something to do with that pool configuration!

Ok let’s check it. The official documentation tells us that:

Active Record database connections are managed by ActiveRecord::ConnectionAdapters::ConnectionPool which ensures that a connection pool synchronizes the amount of thread access to a limited number of database connections.

and

If you try to use more connections than are available, Active Record will block you and wait for a connection from the pool. If it cannot get a connection, a timeout error similar to that given below will be thrown.

Here it is! This is what is happening:

That’s what must happen here!

Ok, then, what about the other side? How is Sidekiq managing its threads to prevent this error? The official wiki is pretty straightforward:

You can tune the amount of concurrency in your sidekiq process. By default, one sidekiq process creates 10 threads.

So, it creates as many threads as we mentioned on our config/sidekiq.yml

concurrency must be the key!

There is also a little note on the wiki:

Note that ActiveRecord has a connection pool which needs to be properly configured in config/database.yml to work well with heavy concurrency. Set the pool setting to something close or equal to the number of threads

THAT’S IT! If we have Sidekiq concurrency value equal to Active Record pool value we can’t have more threads than the accepted by Active Record, and no more errors. Voilà! Let’s check those values on our project, something should not be right.

and

So… is everything ok?

Ok, now things are really strange. On our configurations files we already have 3 for concurrency and 3 for pool fields, then, according to documentations, everything should work and we should not have any issues… But we have one! Then, I don’t know how but somewhere, at a time, 3 looks not to be equal to 3.

Let’s double check everything, somewhere one of those value has to be overridden.

Of course, the answer was NOPE.

Let’s investigate 🕵🏻‍♀️

As the error occurred each day at the same hour, we looked at the piece of code running at that moment and checked the logs.

We analyzed 40 lines that occured in less than 10 seconds. The result is a beautiful file named sidekiq logs sherlock with many background colors and others for the text. And we found something interesting…

Can we reproduce it?

Of course, we can try. Let’s create a tiny Rails app with a database and Sidekiq. Let’s have two configuration files with the same configuration, as we have seen before:

Now we can create a worker:

This is a simple worker, doing nothing but calling a service which is going to do some stuff. For this service, we tried to reproduce the code we have, but this code is calling some other gem. To avoid complexity, we try to simplify it but keeping the main behavior.

Try to run your worker and look at the result…

ruby-2.5.3/gems/activerecord-5.2.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:199:in `block in wait_poll’: could not obtain a connection from the pool within 5.000 seconds (waited 5.010 seconds); all pooled connections were in use (ActiveRecord::ConnectionTimeoutError)

We are not sure that is exactly mimicking our behavior as the code uses transitive dependencies that are creating threads. But it looks like this and we ran into the same issue.

And it makes sense. Remember, we said that for each process Active Record assigns x connections depending on the pool value. Each connection is made by one thread. If our Sidekiq concurrency value is equal to the pool value, we should not have any problem, unless, something is creating a new thread and calling the database. That’s the case on our example above with the MyService class!

The why!

In fact, it’s possible to do async job here but we need to be sure that we are not dealing with database, or at least, it’s not creating too much connections. In our case, we are inside a deep loop, which leads to too many database calls. 💥

If you don’t want to have some headaches, be careful on what you put inside your workers 🙂

Photo by Ayo Ogunseinde on Unsplash

Thank you for reading! 🙏

You should subscribe to our blog to be notified whenever a new article is available! 👋

This article was written by Mélanie Araujo Martins of Appaloosa’s dev team.

Test your iOS application UI

appaloosa.io — Wed, 23 Jan 2019 15:12:54 +0000

Introduction ✍️

In this article, we will explain first how writing UI Test using Xcode, then how we added visual diff and finally how we configured in Bitrise.

Why testing the User Interface (UI)?

For production release, it is painful to test every single use case and be sure there is no regression. You can either forget to test one flow or think that your change will not impact other parts of the code… But sometimes, it does 😣. This is what happened at Appaloosa for our iOS application, where some errors slipped under the radars. For example, when iOS 11 was released, we added constraints in the navigation bar, but this feature was not available in prior versions and the app crashed in iOS 10.

It was quickly identified and fixed, yet that alert encouraged us to have a stronger process to prevent such issues.

UI Test and Unit Test

With unit tests, you test each small piece of code independently and focus on the logical aspects of the code.

With UI tests you simulate user interactions and user flows. Let’s take the example of a user trying to log in your application. It will test the different interactions the user has with our application during the whole process.

Setup 📚

Go to your project on Xcode, click “+” button (see the image below) and add a iOS UI Testing bundle. A new target will be created as well.

You can also add it when you create your project on Xcode.

Sample project

Here is an example project. We want to test multiple user flows and be sure they all work as expected.

You can find the complete code on github.

The goal is to make sure our login page shows the right views during the different steps. So we will need to access the different views of the page to evaluate their behaviour during the test.

You have multiple ways to retrieve an element of your application:

You can use the recording button in Xcode, it will generate automatically your code.
You can add an accessibility identifier on your view to access it more easily.

You can write your code manually, in that case, you need to know how to access an element. You can print the detailed view hierarchy of the visible screen using the following command in LLDB:

(lldb) po print(app.debugDescription)

I prefer this last method, the code is more readable.

Writing your first UI test

Login with invalid credentials:

If a user enters invalid credentials, we expect the application to show an error message:

Let’s see how this translates in code.

The user wants to display his application list:

Here we are forced to wait for the response of our fake server. We need to tell Xcode to wait until the application finished rendering the view. If we don’t, our test will fail since the view component being targeted has not yet appeared.

To force Xcode to wait, I used an extension to XCTestCase:

The user wants to see the details of an application:

Starting from Xcode 9, we can’t compare strings longer than 128 characters anymore so we are forced to use a predicate:

✌️Now you can add UI tests for various user flows!

Visual Diff

UI tests can’t be used to detect design changes (eg: different text size or text color)

At Appaloosa, the frontend team faced the same issue and, after some research, they found a solution for visual regression test called Percy. So I looked for a similar solution for iOS.

Note: Benjamin, our front end expert wrote an article about Percy.

Using Snap.swift 🎨

Snap.swift is a library that can compare a referent image (a screenshot) to the rendered UIView. This library was inspired by a facebook library, which is now maintained by Uber.

This library is usually used for unit tests, so I had to adapt my UI tests.

Setup:

The first change I needed to make is to hide the status bar. If you don’t do this, your tests will fail every time.

To do this, I added the following line of code:

We then need to generate screenshots for every view in the app using the following command:

isRecording = true in your setup method of your UI test.

If something changes on the view, Snap.swift will generate two folders:

Diff
Failure

the diff image will show the difference between the failed image and the expected image.

We now have a functional setup to run UI tests locally, and we want to integrate it to our CI.

Bitrise 🤖

At appaloosa, we use Bitrise for Continuous Integration. Below you can see the flows we use when launching a new production deployment. All flows need to be run successfully for the deployment to be completed.

In order to avoid regression within the application, we decided to run our UITest flow on different screen sizes and iOS versions. At the time of writing, the flow is tested against the iPad, the iPhone 7, the iPhone 6 Plus and the iPhone X on iOS from version 10 to 12.

When UI tests fail, we need to retrieve the screenshots associated to the failed tests (generated by Snap.swift).

To do this, we added a Bitrise flow: Deploy to Bitrise.io — Apps, Logs, Artifacts.

This is also used to create new screenshots or update existing screenshots when a view changes.

Limitations ⏳

Every time the design of the app changes, we need to force tests to fail in order to retrieve the new “valid” screenshots as the new point of reference. This goes against the principles of Continuous Integration but we did not find a suitable alternative (yet).

Thank you for reading! 🙏

If you have any questions or suggestions, please leave a comment we will be glad to answer it!

👋 You should subscribe to our blog to be notified whenever a new article is available!

This article was written by Elise El Beze of Appaloosa’s dev team.

Demystifying git rebase: A workshop

appaloosa.io — Wed, 23 Jan 2019 14:57:28 +0000

Practical guide to learn about the git rebase command with a poem.

Image credit: **professortocat_v2 from **GitHub Octodex

git rebase is the process of reapplying commits on top of another base tip.

Birth of an idea

The git rebase command is quite complex and after several discussions during team meetings we discovered that some of us did not know how to use it properly. We were often avoiding the git rebase command out of fear, since this command can lead to all kinds of trouble if not used properly.

Some of the team members are working with git GUI clients, such as SourceTree or GitKraken, to work with a visual representation of their repositories. Some don’t know advanced use of some commands.

So, we decided it would be a good idea to organize a workshop to learn this all together since most of us had knowledge gaps on this similar subject.

I really wanted this to be less scholar and more fun, so I volunteered to create a hands-on workshop for the team.

I thought about using a poem. “The road not taken”, by Robert Frost.

Two roads diverged in a yellow wood,
And sorry I could not travel both
And be one traveler, long I stood
And looked down one as far as I could
To where it bent in the undergrowth;

Then took the other, as just as fair,
And having perhaps the better claim,
Because it was grassy and wanted wear;
Though as for that the passing there
Had worn them really about the same,

And both that morning equally lay
In leaves no step had trodden black.
Oh, I kept the first for another day!
Yet knowing how way leads on to way,
I doubted if I should ever come back.

I shall be telling this with a sigh
Somewhere ages and ages hence:
Two roads diverged in a wood, and I —
I took the one less traveled by,
And that has made all the difference.

Preparation

I simulated the work of a team contributing to the same poem, like so:

Each branch corresponded to a poem’s paragraph or a small part.

The different branches contained bad commits (in red), and commits to rearrange (in yellow).

The objective of the workshop was to recompose the full poem branch after branch, using the git rebase command. Commands, such as the git cherry-pick command, were forbidden.

When creating the workshop, one commit corresponded to only one line of the poem.

“P1/L3” was the line of the poem from paragraph 1 line 3: “And be one traveler, long I stood”.

The objective was to make it easier to resolve conflicts. They often occur when changes are made to the same line of a file.

To resolve a merge conflict, we usually have to choose which changes to incorporate, so I indicated the number of the line for each line of the poem (see below).

The poem was an excuse to use the git rebase command. The workshop was not focused on conflict resolutions.

The workshop

Each member of the dev team set up their environment, cloning the repository:

https://github.com/coralieco/workshop-git-rebase

This workshop is a practical guide to train on a complex command using git rebase. Everyone was allowed to read the documentation or to ask help live. It was not a competition, but the workshop had to be completed in one hour or less :)

We started the workshop together, rebasing the first paragraph as a first example. This was to make sure everyone understood the rules and the methods to go along with the rest of the workshop.

Then, everybody had to complete the workshop. Using the git rebase command, and more particularly the — interactive option.

From documentation:

With the interactive rebase tool, you can then stop after each commit you want to modify and change the message, add files, or do whatever you wish. You can run rebase interactively by adding the -i option to git rebase. You must indicate how far back you want to rewrite commits by telling the command which commit to rebase onto.

There is one option I particularly liked, so I suggested to the team to use it to rebase the poem into paragraph 4 without rebasing it into the parent paragraph (see diagram below) : git rebase -onto

git rebase documentation)

As there was different levels of progression in the team, I corrected one paragraph every 10–15 minutes approximately, so everyone could follow.

At the end, I sent a detailed correction of the commands I used to reconstitute the poem. Some of the dev team members felt the need to redo the workshop at home in order to really benefit from this exercise.

Conclusion

Git rebase interactive gives us the opportunity to alter individual commits.
Git rebase is great to maintain a linear project history, clean and meaningful, we must not be afraid to use it :)
Practical workshops are really successful to learn a new subject: we were not petrified to use it on Appaloosa project.

Overall, we were all happy of the format. We all enjoyed spending this time together and we would like to continue organizing similar workshops in the future.

Thanks to Appaloosa Team!
This article was written by Coralie Collignon of Appaloosa’s dev team.

How to slowly replace your site with a new one made with EmberJS

appaloosa.io — Wed, 23 Jan 2019 14:53:55 +0000

As you may know (if not, read our articles 😉) at Appaloosa we recently release our new frontend to our customers (read We have a new EmberJS front end).

More than two years ago, we started the work on this new frontend which is a complete redesign of our old massive frontend (lot of features/pages). At the time there’s still works to do, the new admin does not cover the global spectrum of our old admin. But we decided to release to our customers, because it’s a nice interface with a better UX and it’s easier to update/maintain.

For the parts that are not available (yet), we redirect our users to our old/legacy admin.

This new frontend is build with EmberJS and today we decided to show you how we manage the transition from our new admin to our old admin. We decided to share our solution, because we think it’s a good one that might help people that need to add the same behavior.

But first let’s have a small look of how it looks in real.

Looks great, isn’t it ?

Why a redirection modal?

Before dealing with the redirection, let’s create a nice modal. The modal is very important because it inform the user about what’s happening. Without the modal, the user would be lost. Imagine being are on a nice looking interface and suddenly ending up on another website with a very different and old-looking design 😱.

How to make a redirection

Let’s start with some HTML

Now that we have a modal, let’s have some CSS to make a good looking modal.

Now that we have a nice modal, let’s make the redirection. To make it easy to add/remove redirection in our templates we’ve decided to go with a service and use a simple html link.

Let’s have a look to the redirection service.

This service is pretty simple, it consist on 3 actions :

redirect: This action setup the redirection, by saving the request url, and displaying the redirection modal.
proceedRedirection: This action triggers the redirection to the requested page
cancelRedirection: Cancel the redirection. Cleanup the redirection service by erasing the saved url and close the redirection popup.

It also expose displayModal that is use to display the modal when needed.

Because with Ember you can specify the target for an action. Adding a redirection in the app is as simple as adding a link tag in a template (component or route). The only requirement is to inject the service.

As you can see it’s pretty easy to add such redirection service the only tricky part is to detect back navigation and close the redirection popup. Fortunately since Ember 2.15 there’s a router service you can use (see the release note here). Using this service you can easily detect back navigation by observing the router’s current route. So let’s change the previous code a little bit.

Now we’re able to detect any “back” navigation action and close the modal automatically.

Doing more than redirection

In this article we share how to create a simple redirection which opens a popup, it’s just a basic simple example there’s more you can do.

For example, at Appaloosa we send some analytics when a user is redirected. That way we prioritise the transfer of a feature from the old admin to the new one, because we know what feature our users need the most.

With the same code you can also add an input field inside the redirect popup asking for a feedback, or rate their experiences… is up to you 😉

👋 You should subscribe to our blog to be notified whenever a new article is available!

This article was written by Benjamin JEGARD of Appaloosa’s dev team.
PS: There’s a live example of this redirection service here 🤩.

Ruby memory, ActiveRecord and Draper

appaloosa.io — Wed, 23 Jan 2019 14:42:23 +0000

At Appaloosa we often process CSV. When we process thousands of CSV lines of users that need to be added to (or removed from) a group, we are seeing a huge increase in memory. Sometimes the Heroku dynos cannot manage it.

We will explain how we jump from this when processing 4_000 users (look at blue line but also time):

To that:

memory profiler

We used stackprof, memory_profiler. We look at tool like speedscope for stackprof results and always it point at heavy use of ActiveRecord. Also we noticed in “Allocated String Report” of memory_profiler a string allocated multiples times.

With 4000 users:

Allocated String Report
-----------------------------------
   1619100  "decorated?"
   1619100  .rvm/gems/ruby-2.5.3@app/gems/activerecord-5.2.1/lib/active_record/attribute_methods.rb:279

"decorated?" where does it come from?

It’s instantiated here: .rvm/gems/ruby-2.5.3@app/gems/activerecord-5.2.1/lib/active_record/attribute_methods.rb:281

If we add a puts caller when "decorated?" is instantiated we have this:

First we didn’t understand how we can jump from Draper to active_record/associations/collection_association.rb like:

The code on Draper part is:

Then the code in active_record/associations/collection_association.rb:281 looks like:

We jump from index = @target.index(record) to Draper::Decoratable::Equality.== 🤔.

But how?

Christophe Maximin (thanks a lot, Christophe) came with this answer:

(Wild guess by just reading the AR code) Because if the same user (same primary key) is already in users, it just replaces that old object with the new object you’re passing with <<, which is why it needs to know the index of the old one. And to know if records are the same, it needs to call ==.

If you want a full example of how it is called, you can use [TracePoint](https://ruby-doc.org/core-2.5.3/TracePoint.html). Without and with having == in Integer:

Tracepoint “Establishes proc as the handler for tracing” calls :call: for call to Ruby method :c_call: for a call to a C-language routine

What happens on a real app?

To be sure we were not testing a side effect of our app, we created a new rails app, with 2 models, Draper installed. Then we monkey patched add_to_target for ActiveRecord.

We put the code to reproduce the concatenation in a rake task.

If we add two records rails concat_records:comments[2] on the second record we have this interesting result:

We can see that we are calling index, then Draper::Decoratable::Equality (that override ==), then ActiveRecord:Core, etc...

The issue is when the array of record we concat contains lot’s of ActiveRecord objects. Every time we add one element, we redo all of those calls. Also Draper in is Equality method add more calls, and one of the call adds :decorated? string. All of this chain have a cost.

Let’s measure it.

Measuring allocation with Draper, without Draper, without concatenation

With Draper (branch)

Draper gem installed. No decorators defined but == is overridden by Draper as we can see.

If we run a measure with memory_profiler: rails concat_records:comments[2000]. We get:

Total allocated: 327.93 MB (4862043 objects)

Total retained:  17.67 MB (172370 objects)

And we can see also:

Allocated String Report

-----------------------------------

1999000  "decorated?"

1999000  .rvm/gems/ruby-2.5.3/gems/activerecord-5.2.1/lib/active_record/attribute_methods.rb:282

Without Draper (branch)

First we don’t have any more the very present allocated string "decorated?". First allocated string is ”” with 40185 occurrences. That’s a gain of 1 958 815 strings.

Allocated String Report

-----------------------------------

40185  ""

24000  .rvm/gems/ruby-2.5.3/gems/activesupport-5.2.1/lib/active_support/tagged_logging.rb:23

If we run a measure with memory_profiler rails concat_records:comments[2000]. We get:

Total allocated: 247.97 MB (2863011 objects)

Total retained:  17.67 MB (172370 objects)

It’s little bit better.

Without concat but with Draper (branch)

Same as before without Draper we don’t have any more very present allocated string "decorated?":

Allocated String Report

-----------------------------------

40185  ""

24000  .rvm/gems/ruby-2.5.3/gems/activesupport-5.2.1/lib/active_support/tagged_logging.rb:23

And…

Total allocated: 218.73 MB (2377978 objects)

Total retained:  31.95 kB (284 objects)

This is a huge gain but we don’t validate that we don’t create a duplicated entry into the join table. But this is something easy to handle.

How this can happening?

If their is a distinct on the association scope like this:

Then when ActiveRecord needs to concatenate. It will call @target.index(record). If @target have a lot's of elements this will consume more some CPU and memory.
And if Draper is installed this will add extra cost because Draper override == called from index. This override will allocate extra strings we don't need ("decorated?").

Looking in index plus using Draper is a logarithmic cost for iteration per second and memory. More element you have in "target" slower it gets.

So careful when using distinct in a scope. Also careful with Draper.

Here is a diagram of what is happening

Diagram of index search and Draper override

In our code base the patch looks like:

Thanks

Thanks to Appaloosa Team, Christophe Maximin, Ryuta Kamizono (kamipo), Nate Berkopec and the Rails Speed Community Slack.

👋

You should subscribe to our blog to be notified whenever a new article is available!

This article was written by Benoit Tigeot of Appaloosa’s dev team.