DEV Community: Cédric Fabianski

How Rust Lets Us Monitor 30k API calls/min

Cédric Fabianski — Wed, 08 Jul 2020 08:40:51 +0000

At Bearer, we are a polyglot engineering team. Both in spoken languages and programming languages. Our stack is made up of services written in Node.js, Ruby, Elixir, and a handful of others in addition to all the languages our agent library supports. Like most teams, we balance using the right tool for the job with using the right tool for the time. Recently, we reached a limitation in one of our services that led us to transition that service from Node.js to Rust. This post goes into some of the details that caused the need to change languages, as well as some of the decisions we made along the way.

A bit of context

We are building a solution to help developers monitor their APIs. Every time a customer’s application calls an API, a log gets sent to us where we monitor and analyze it.

At the time of the issue, we were processing an average of 30k API calls per minute. That's a lot of API calls made across all our customers. We split the process into two key parts: Log ingestion and log processing.

We originally built the ingestion service in Node.js. It would receive the logs, communicate with an elixir service to check customer access rights, check rate limits using Redis, and then send the log to CloudWatch. There, it would trigger an event to tell our processing worker to take over.

We capture information about the API call, including the payloads (both the request and response) of every call sent from a user's application. These are currently limited to 1MB, but that is still a large amount of data to process. We send and process everything asynchronously and the goal is to make the information available to the end-user as fast as possible.

We hosted everything on AWS Fargate, a serverless management solution for Elastic Container Service (ECS), and set it to autoscale after 4000 req/min. Everything was great! Then, the invoice came 😱.

AWS invoices based on CloudWatch storage. The more you store, the more you pay.

Fortunately, we had a backup plan.

Kinesis to the rescue?

Instead of sending the logs to CloudWatch, we would useKinesis Firehose. Kinesis Firehose is basically a Kafka equivalent provided by AWS. It allows us to deliver a data stream in a reliable way to several destinations. With very few updates to our log processing worker, we were able to ingest logs from both CloudWatch and Kinesis Firehose. With this change, daily costs would drop to about 0.6% of what they were before.

The updated service now passed the log data through Kinesis and into s3 which triggers the worker to take over with the processing task. We rolled the change out and everything was back to normal... or we thought. Soon after, we started to notice some anomalies on our monitoring dashboard.

We were Garbage Collecting, a lot. Garbage collection (GC) is a way for some languages to automatically free up memory that is no longer in use. When that happens, the program pauses. This is known as a GC pause. The more writes you make to memory, the more garbage collection needs to happen and as a result, the pause time increases. For our service, these pauses were growing high enough that they caused the servers to restart and put stress on the CPU. When this happens, it can look like the server is down—because it temporarily is—and our customers started to see 5xx errors for roughly 6% of the logs our agent was trying to ingest.

Below we can see the pause time and pause frequency of the garbage collection:

In some instances, the pause time breached 4 seconds (as shown on the left), with up to 400 pauses per minute (as shown on the right) across our instances.

After some more research, we appeared to be another victim of amemory leak in the AWS Javascript SDK. We tried increasing the resource allocations to extreme amounts, like autoscaling after 1000 req/min, but nothing worked.

Possible solutions

With our backup plan no longer an option, we moved on to new solutions. First, we looked at those with the easiest transition path.

Elixir

As mentioned earlier, we are checking the customer access rights using an Elixir service. This service is private and only accessible from within our Virtual Private Cloud (VPC). We have never experienced any scalability issues with this service and most of the logic was already there. We could simply send the logs to Kinesis from within this service and skip over the Node.js service layer. We decided it was worth a try.

We developed the missing parts and tested it. It was better, but still not great. Our benchmarks showed that there were still high levels of Garbage Collecting, and we were still returning 5xx to our users when consuming the logs. At this point, the heavy load triggered a (now resolved) issue with one of our elixir dependencies.

Go

We considered Golang as well. It would have been a good candidate, but in the end, it is another Garbage Collected Language. While likely more efficient than our previous implementation, as we scale there is a high chance we'd run into similar problems. With these limitations in mind, we needed a better option.

Re-architecting with Rust at the core

In both our original implementation and our backup, the core issue remained the same: garbage collection. The solution was to move to a language with better memory management and no garbage collection. Enter Rust.

Rust isn't a garbage-collected language. Instead, it relies on a concept called ownership.

Ownership is Rust’s most unique feature, and it enables Rust to make memory safety guarantees without needing a garbage collector.

— The Rust Book

Ownership is the concept that often makes Rust difficult to learn and write, but also what makes it so well suited for situations like ours. Each value in Rust has a single owner variable and as a result a single point of allocation in memory. Once that variable goes out of scope the memory is immediately returned.

Since the code required to ingest the logs is quite small, we decided to give it a try. To test this we addressed the very thing that we had issues with—sending large amounts of data to Kinesis.

Our first benchmarks proved to be very successful.

From that point, we were pretty confident that Rust could be the answer and we decided to flesh out the prototype into a production-ready application.

Over the course of these experiments, rather than directly replacing the original Node.js service with Rust, we restructured much of the architecture surrounding log ingestion. The core of the new service is an Envoy proxy with the Rust application as a sidecar.

Now, when the Bearer Agent in a user's application sends log data to Bearer, it goes into the Envoy proxy. Envoy looks at the request and communicates with Redis to check things like rate limits, authorization details, and usage quotas. Next, the Rust application running alongside Envoy prepares the log data and passes it through Kinesis into an s3 bucket for storage. S3 then triggers our worker to fetch and process the data so Elastic Search can index it. At this point, our users can access the data in our dashboard.

What we found was that with fewer—and smaller—servers, we are able to process even more data without any of the earlier issues.

If we look at the latency numbers for the Node.js service, we can see peaks with an average response time nearing 1700ms.

With the Rust service implementation, the latency dropped to below 90ms, even at its highest peak, keeping the average response time below 40ms.

The original Node.js application used about 1.5GB of memory at any given time, while the CPUs ran at around 150% load. The new Rust service used about 100MB of memory and only 2.5% of CPU load.

Conclusion

As with most startups, we move fast. Sometimes the best solution at the time isn't the best solution forever. This was the case with Node.js. It allowed us to move forward, but as we grew we also outgrew it. As we started to handle more and more requests, we needed to make our infrastructure evolve to address the new requirements. While this process started with a fix that merely replaced Node.js with Rust, it led to a rethinking of our log ingestion service as a whole.

We still use a variety of languages throughout our stack, including Node.js, but will now consider Rust for new services where it makes sense.

How to make remote a success

Cédric Fabianski — Tue, 14 Jan 2020 15:44:10 +0000

A couple of weeks ago, I attended the P9 Founder Summit in Malta organized by Point Nine Capital. One of the workshops was about remote work.

Soon after, I led a webinar for the #p9family and thought I would share our experience at Bearer thus far.

Liquid error: internal

It's all about sharing and communicating

When you are small and in the same office, it's pretty easy to be aware of everything. The whole team spends a lot of time hanging out together—having lunch, grabbing a coffee, having discussions, and making decisions very quickly. As you scale up, more discussions take place, more decisions get made, and it becomes harder to keep up. You might even become successful enough that your office expands to a second floor or even a new branch in a different country.

At this point, processes are put in place to improve that internal communication. They are best practices that everybody ends up doing, but often after the fact. When you are remote-first, those processes need to be put in place from day one.

I've been lucky enough to be a remote worker for most of my career. Even my first internship was as a remote worker! My last experience at FreeAgent taught me a lot, and I'm applying many of those insights here at Bearer.

Remote as a first-class citizen

I, as Co-Founder and CTO, am myself remote most of the time. I'm visiting the Paris offices for customer meetings, board meetings, conferences, and so on 2 days every 2 weeks on average. I think it's very important for people to see that remote is something we really believe in and view us as an example of how it can be done right. I've attended TechRocks and during one of the talks, Julien Dollon, Director of Engineering at Oracle gave this tip:

If you want remote to work, start by cutting off the head

For him, this meant being the first to work remotely, while the rest of the team stayed in the office.

When the chief is remote, all of a sudden everybody is remote and everybody starts writing down everything. What used to be ephemeral and on a whiteboard became written down and stored.

That's what I saw at FreeAgent too and I think that's an important part of the process that shouldn't be neglected.

Write down everything!

Select a shared language. For us, that's English. All public communication needs to be in English. Even as French founders, this is something we have done from day one and that means that anybody can look back at why we've made a particular decision.

Use Requests for Comments (RFCs). The Open Source community has a lot of experience in being remote-first. The need to share information while working asynchronously from different timezones is no secret. Every decision made within the project needs to be documented and approved. When someone looks back on it later, they can understand what led to a particular technical decision. Here are a few examples:

Build a Knowledge Base. Having a meeting? Learning something new? Retrospective? How-Tos? These need to be documented and shared! My not-so-secret goal is that ultimately, everything that is in the knowledge base will be turned into a Blog Post like what we did with Understanding Auth Part 1: What is OAuth 2.0.

Every discussion longer than 10 messages in Slack needs to trigger a call in Zoom and a summary in Slack or the Knowledge Base.

Define your Culture, Vision & Mission. Every employee should be able to easily find the Culture, the Vision, and the Mission of the company. It keeps everyone working toward the same, shared goal and reminds us all what matters most as a company.

Organize Weekly Demos. Every week on Thursday at 5 p.m. CET we gather together and answer these questions:

- What are the numbers this week?
- What has been done company-wise?
- What has been done product-wise?

It's recorded and pushed to YouTube. If someone is off, in a timezone that makes it harder to join, or would like to come back to it later, they can watch it.

Send Out Weekly Notes. At the beginning of the week, send out a note to the team that recaps the week before and the goals for the coming week. Use the weekly demo as a foundation, and cover the following:

- News of the week
- Goals of the week
- Roadmap this week
- Done last week
- Team event (who is off, who is where)
- Link to the recording from the weekly Demos

Make everyone feel connected

Connect in person. As a remote-first company, we spend most of our time reading our colleagues rather than hearing them. Some team members aren't native English speakers, and sometimes if you don't know the person well it can be easy to misinterpret the intent from their writing alone.

That's why we believe in getting together in-person at off-sites. The last one took place in Portugal.

Liquid error: internal

This also means that you shouldn't expect to spend less money by setting up a remote company if you want to do it right.

Liquid error: internal

Share a coffee break. When you are all in an office, you have those coffee machine moments. You discuss things like what you did last weekend, what you are up to outside of work, and so on. These moments are an important part of getting to know one another. At Bearer, we put one on the calendar each week, but I also encourage the team to open up a meeting in our #coffee channel whenever they like.

Conduct smarter meetings. Meetings are expensive. Not just in the time they take, but also in the disruption of everyone's workflow before and after. Make sure to use this time wisely.

Before a meeting, send out an agenda to give everyone a chance to contribute. Whether that is an RFC, a Google Doc or a Notion note, there needs to be something sent beforehand. After the meeting, summarize in the outcome and list the actions that should come from it.

Perform daily check-ins/check-outs. It's important to share your progress with the team. Usually, that happens during a daily standup. While this is not mutually exclusive, some squads at Bearer do catch up every morning. What started as a squad initiative is now part of the Product & Engineering routine. We use Geekbot to allow each team member to answer the following questions each day:

- How did you feel today?
- What have you done today?
- What's your plan for tomorrow?
- Have you been blocked by anything?
- What have you learned today?
- How efficient did you feel today? (1-5)

These questions help us measure not only productivity but also the spirit and morale within the Team. We also encourage people to publish their learnings of the day by answering the question:

What have you learned today?

Tools we use

Like other remote companies, we rely on many tools to help with all of this. Here are a few that we've had success with:

Zoom and Slack are a great foundation to start with. You can trigger a call within Slack using this command /zoom and jump directly into a call for a pairing session, right from where you are.

Geekbot to trigger the checkin/checkout and to get a feeling about how the team is doing.

Notion is basically used as an entry point at Bearer. Everything you need to know is already there. GitHub is used for our code and as such is a great tool to keep track of the technical decisions (RFCs).

Geckoboard and Redash are great to share the metrics to the Team. It's important that everybody has access to the same level of information that drives business decisions.

Miro is a great replacement for whiteboard and has the advantage that you can get back to it at any point without the fear of erasing somebody's else diagram by mistake.

Clocker is a menu bar on macOS and that shows you what time it is in everyone's locations.

We're also considering a few more like OfficeVibe and 15Five to measure the Team happiness which is really important to us.

There's always more that can be done

We are only at the beginning of our journey. We will surely make mistakes along the way, but we will continue to learn from them.

Something that is working today may not work tomorrow as your company grows. You will need to adapt, and I'll continue to share our processes around and commitments related to remote work here at Bearer.

For instance, right now, we are mainly in the European Timezone and we ask people to have at least 6 hours overlap within that Timezone. That's a decision we've made intentionally to avoid loneliness and isolation. We also feel like that's the best way to be effective, at least at our stage. This might change later as more team members join in other timezones.

Share your remote tips in the comment below 👇.

PS: This post was originally published on Bearer blog.

How we got 900 applications on our Developer Position

Cédric Fabianski — Tue, 29 Oct 2019 16:41:42 +0000

Bearer is a remote-first company; we have always found this to be a strength, for many reasons.

Hiring is one of them.

Thus far, we hired, mainly opportunistically, by networking. This method worked well up to the point when we needed to expand our circle and hire from the outside.

From 1 to 10

Guillaume and I have been in the industry for a while now; we've met many developers. That's providing us an unfair advantage. When we open a position, we naturally discuss it with those within our network. That's how we got to be a team of 10!

The jobs page we created in the beginning has also helped us source outside talent. Additionally, we also tweet about opportunities on the Bearer Twitter account. From time to time, we even receive emails from people, prospecting.

Up to this point, we've only been onboarding Senior Engineers because we felt like we didn't have the structure to support a Junior Engineer. In the past few months, we've made a lot of progress and it was time to take a step back and re-evaluate this position. We've just rebuilt our dashboard from the ground up (built with TypeScript, ReactJS, GraphQL) and the foundations are rock solid (we will blog about this 😉). The website has also proven to be future-proof and easy to maintain. We were ready to onboard a new Engineer and this time it would be a Junior!

We published a job offer for a Junior/Mid-Level, Frontend Engineer and were inundated by applications.

10 and beyond

We got in touch with the amazing people at remotive.io and were convinced that their platform would be perfect for us, a remote-first company. In order to increase our chances of finding the perfect gem, we also decided to post the opportunity on GitHub Jobs and JSRemotely too.

Things were simple, I created an alias to receive the job applications. This alias redirected enquiries to Antoine, our Senior Frontend Engineer, and me; he was to be in charge of the first steps, selection and the first interview.

We understood, pretty quickly, that we were not enough prepared.

After 3 days, we had received 250+ applications; after 1 week, 500+; and after 2 weeks, we were at ~900. This is without including the LinkedIn and Twitter messages.

Time to investigate some real processes

After a couple of days, Antoine and I spoke; we were both excited and overwhelmed. We knew that we couldn't review every application. We needed help.
A week later, I'd put together a survey, using Typeform. It would ask some of the questions we usually asked during an interview:

Do you know Gatsby, ReactJS, Styled Components, GraphQL, TypeScript, Tailwind CSS, ...?
Are you willing to work in the European Timezone?
What's your GitHub handle?

I then found a way to send this Typeform to all the existing applicants as well as the new. Within a week it started to get responses; this helped us a lot.

My mailbox was being filled with messages like:

I've filled the Survey

Thanks for getting in touch

But at least we could keep track of them all and could filter applicants to meet our needs.

In total, we got 430 responses.

The standard of applicants was impressive, and we were lucky enough to make the final choice from amongst these amazing candidates, from all over the world!

Lessons Learned

Reply to everyone

One thing I've noticed, is that there is nothing more frustrating than not being given feedback after submitting an application. We tried to reply as fast as possible, to the survey after 1 week and we sent rejection emails after 2.

Get ready for applications

Set up a Typeform, that will be your first line of defense. Get candidates to go straight to it and answer your very first questions there. Have the replies sent directly to something like Google Sheets or Airtable (there is an integration for both in Typeform).

Share and be transparent

We have a blog; our blog post, sharing how we work and our company culture, has been looked at quite a lot in the past few days. Sharing and being transparent is very important for candidates. They want to know as much as possible about you before they apply.

Remote is great to get the best candidates

We are still relatively unknown and yet we received quite a lot of applications. We've been really impressed by the levels of engagement of some candidates.

Working on a developer tool definitely helps

Most candidates were really interested in joining because we were building a developer tool that they found interesting. Some got back to us with an example of applications they have built with Bearer using Google Spreadsheet API for instance. This is definitely an unfair advantage to attract talented developers.

Conclusion

Out of this, we hired Rien Coertjens, at first with a short and specific mission. We have been very happy with him and we decided to hire Rien full-time.

Of course, our process is far from being perfect; we may have rejected some great candidates who would have been fantastic fits for us. Some applicants may not have understood the reasons why they didn't make it to the next cut, and I'm really sorry about this. We had to make some tough decisions, but it was simply not manageable otherwise.

But...

We will be opening more developer positions in the future.

Discuss this article on Twitter and ping us @BearerSH