DEV Community: Thijs Cadier

CPU Steal Time: A Crucial Metric for Cloud Servers and VMs

Thijs Cadier — Thu, 23 Sep 2021 11:02:58 +0000

If your application runs in a virtualized environment, there is a crucial metric you might not be aware of: CPU steal. In this post, we'll explain what CPU steal is, how to monitor it, and what happens to your app when CPU steal is high.

What Is CPU Steal?

CPU steal is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor. It gives you an indication of which percentage of a real core you can use in a given moment. This metric is valuable for those who run their infrastructure in the cloud, e.g. AWS, DigitalOcean, etc.

CPU Steal in a Real-life Example

Let's say you have a server running on DigitalOcean. For the sake of this example, your server runs on a physical machine that hosts 5 VMs. All of those VMs share the same resources. Memory usage is capped at 20%, but CPU cycles aren't. This means that one VM can use more than others.

If you SSH into your server and run the top command, the CPU steal time will be marked as st.

top - 13:26:09 up 22:38,  1 user,  load average: 1.90, 1.92, 1.60
Tasks: 373 total,   1 running, 181 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.8 us,  0.3 sy,  0.0 ni, 94.5 id,  0.4 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 13186532+total,   810704 free, 52079712 used, 78974904 buff/cache
KiB Swap:  7811068 total,  7748604 free,    62464 used. 78530328 avail Mem

Let's examine the CPU line here. The last metric indicates the time stolen from this VM by the hypervisor. CPU waits for this VM's hypervisor to attend to another VM.

If steal time is low, it means that VMs aren't busy, but this is rarely the case with web applications that serve users all over the globe. If a VM you're sharing the server with has a task running for a long time in the background, that one is highly likely to get more than their fair share of CPU cycles for a while. Afterwards, other VMs will jump in the queue and slow down that task.

These tasks hurt web applications the most - they can cause performance issues and even lead to outages.

What to Do about CPU Steal Time

Let's run through a few examples.

CPU Steal Time Is Low

If your CPU steal time is lower than 10%, you have nothing to worry about. Your application should run smoothly.

CPU Steal Time Is Relatively High

Your CPU steal time is well over 10% for ~30 minutes.

Check what type of virtual server you're running. How fast is the processor, and how much RAM and disk space do you have? There is a possibility that your app needs more CPU resources overall. In that case, you are causing the problem.

However, what if you're already spending a lot of money on the right servers for your web application? In this case, you have every right to complain to your server provider. They are putting more VMs on physical machines, and your noisy neighbors are competing for resources with you.

Beware: there's a catch. You won't know which situation you're in unless you have the full host overview. To test whether you're the cause of the issue, you should run multiple identical VMs that perform the same tasks on several different hosts. If all VMs have issues, you're the one causing them, but if only one VM complains, that means your provider is the one to blame.

How to Monitor CPU Steal with AppSignal

AppSignal displays CPU steal in the host metrics screen for every host by default for Ruby, Node.js, and Elixir applications. In addition, it can be useful to set up a dashboard of just the steal metric if you have infrastructure that's likely to be effected.

All you need to do is add AppSignal to your app, and if you're already using us, upgrade to the latest version of our integration or the standalone agent.

Creating a custom dashboard is a great way to get a focused view on a specific metric like this. I've created a dashboard you can import. Click "Add dashboard", then "Import dashboard" and copy-paste the following dashboard configuration:

{
  "title": "CPU Steal",
  "description": "",
  "visuals": [
    {
      "title": "CPU Steal",
      "line_label": "%hostname%",
      "display": "LINE",
      "format": "percent",
      "draw_null_as_zero": true,
      "metrics": [
        {
          "name": "cpu",
          "fields": [
            {
              "field": "GAUGE"
            }
          ],
          "tags": [
            {
              "key": "host_metric",
              "value": "*"
            },
            {
              "key": "hostname",
              "value": "*"
            },
            {
              "key": "state",
              "value": "steal"
            }
          ]
        }
      ],
      "type": "timeseries"
    }
  ]
}

Let's see some examples of how your dashboard might look.

Low CPU Steal with Some Outliers

This is an example of a healthy situation. Most servers are below 1%.

Sometimes a neighbor is noisy, and you see some outliers.

Consistently Higher CPU Steal

This is an example of a situation where the CPU is oversubscribed. A big chunk of the hosts have very high steal percentages.

Contact your provider's support department to ask if they can look at the capacity allocated to your servers in this case.

What Did We Learn?

If you host in a virtualized environment and see performance issues you can't put your finger on, take a look at the CPU steal metric.

Monitor Your Hosts and Get Stroopwafels

If you haven't had the chance to try AppSignal monitoring, here's what you need to know:

Host monitoring is included alongside all of our features.
We have a free trial option that doesn’t require a credit card.
AppSignal supports Node.js, Ruby, and Elixir projects.
We’re free for open source & for good projects.
We ship stroopwafels to our trial users on request.

Need we say more? 🍪

The Citadel Architecture at AppSignal

Thijs Cadier — Thu, 16 Apr 2020 16:15:38 +0000

DHH just coined the term "Citadel," which finally gives us an excellent way to reference how we approach tech at AppSignal. We said, "Hey, this is us! Our thing has a name now".

In addition to the Majestic Monolith, someone should write up the pattern of The Citadel: A single Majestic Monolith captures the majority mass of the app, with a few auxiliary outpost apps for highly specialized and divergent needs.
— DHH (@dhh) April 7, 2020

To explain how AppSignal uses the Citadel pattern, we'll share a bit on how our system works. AppSignal is a monitoring product that has a user-facing application and an API that the monitoring agent sends data to. This data is then processed and turned into graphs and insights.

Monolith

The application our customers interact with is a monolithic Rails app, with parts of the front-end written in React. The backend is entirely written in Ruby and talks to a few databases (we split out data from different customers to separate clusters for scaling reasons). This Rails app also handles a bunch of other tasks such as sending out alerts to external services.

When we started this Rails app processed incoming data from our monitoring agent as well, we foresaw that data ingestion would turn out to be a bottleneck. So we used a Sinatra app running on a subdomain that ingested data and created Sidekiq jobs that were processed by the Rails app.

Growing Pains

This architecture worked well for years. As our business grew, it became clear that the specific task of processing incoming data from the agents was going to need special treatment. When you're monitoring billions and billions of requests, you run into hard limits. The main limiting factor wasn't so much that Ruby is slow (we all know that it is not 😉), but that the way we had architected things caused too much locking in our databases.

An Outpost

We looked at several possibilities and then decided Kafka was the best fit for our situation. We had some experience with Rust and thought that its speed and reliability would be a very good fit for this system. We rewrote our data ingestion and processing system in Rust, using Kafka as a combination of queue and storage system.

We only moved the incoming data processing part of the Rails app to this outpost service. The rest of the systems works well in the form of a monolithic app. We understand it deeply, and we like keeping things simple. The monolith still handles most of the logic and interacts with Kafka heavily. Our wish to keep our monolith led to us writing a Kafka gem, so the main app can communicate with the outpost easily.

If you want to learn more about how Kafka works at AppSignal check out the Railsconf talk I gave about this.

Life at the Citadel

This brings us to the current situation where we are very happy in our citadel. As DHH said:

A single Majestic Monolith captures the majority mass of the app with a few auxiliary outpost apps for highly specialized and divergent needs.

In our case, we have a single outpost service for our highly specialized needs. If there were a RailsConf this year, we would have given DHH some extra stroopwafels as appreciation for giving it a name. 🍪

Building a Ruby C Extension From Scratch

Thijs Cadier — Tue, 30 Oct 2018 13:50:24 +0000

In this edition of Ruby Magic, we'll show you how to use code written in C from Ruby. This can be used to optimize performance sensitive parts of your code or to create an interface between a C library and Ruby. This is done by creating extensions that wrap libraries written in C.

There are a lot of mature and performant libraries written in C. Instead of reinventing the wheel by porting them we can also leverage these libraries from Ruby. In this way, we get to code in our favorite language, while using C libraries in areas where Ruby isn't traditionally strong. At AppSignal, we've used this approach in developing the rdkafka gem.

So let's see how one can approach this. If you want to follow along and experiment yourself, check out the example code. To start off, let's take this piece of Ruby code with a string, a number and a boolean (you'll C why, pun intended) and port it to a C library:

module CFromRubyExample
  class Helpers
    def self.string(value)
      "String: '#{value}'"
    end

    def self.number(value)
      value + 1
    end

    def self.boolean(value)
      !value
    end
  end
end

In order, the methods shown concatenate a string, increase a number by one and return the opposite of a boolean, respectively.

Our Library Ported to C

Below, you can see the code ported to C. The C Standard Library and the IO Library are included so that we can use string formatting. We use char* instead of a Ruby String. char* points to the location of a buffer of characters somewhere in memory.

# include <stdlib.h>
# include <stdio.h>

char* string_from_library(char* value) {
  char* out = (char*)malloc(256 * sizeof(char));
  sprintf(out, "String: '%s'", value);
  return out;
}

int number_from_library(int value) {
  return value + 1;
}

int boolean_from_library(int value) {
  if (value == 0) {
    return 1;
  } else {
    return 0;
  }
}

As you can see, you have to jump through some hoops to do simple string formatting. To concatenate a string, we first have to allocate a buffer. With this done, the sprintf function can then write the formatted result to it. Finally, we can return the buffer.

With the code above, we already introduced a possible crash or security issue. If the incoming string is longer than 245 bytes, the dreaded buffer overflow will occur. You should definitely be careful when writing C, it's easy to shoot yourself in the foot.

Next up is a header file:

char* string_from_library(char*);
int number_from_library(int);
int boolean_from_library(int);

This file describes the public API of our C library. Other programs use it to know which functions in the library can be called.

The 2018 Way: Use the `ffi` Gem

So, we now have a C library that we want to use from Ruby. There are two ways to wrap this C code in a gem. The modern way involves using the ffi gem. It automates many of the hoops we have to jump through. Using ffi with the C code we just wrote looks like this:

module CFromRubyExample
  class Helpers
    extend FFI::Library

    ffi_lib File.join(File.dirname(__FILE__), "../../ext/library.so")

    attach_function :string, [:string], :string
    attach_function :number, [:int], :int
    attach_function :boolean, [:int], :int
  end
end

For the purpose of this article, we're also going to explain how to wrap the C code with a C extension. This will give us much more insight into how it all works under the hood in Ruby.

Wrapping our Library in a C Extension

So we now have a C library we want to use from Ruby. The next step is to create a gem that compiles and wraps it. After creating the gem, we first add ext to the require_paths in the gemspec:

Gem::Specification.new do |spec|
  spec.name          = "c_from_ruby_example"
  # ...
  spec.require_paths = ["lib", "ext"]
end

This informs Rubygems that there is a native extension that needs to be built. It will look for a file called extconf.rb or a Rakefile. In this case, we added extconf.rb:

require "mkmf"

create_makefile "extension"

We require mkmf, which stands for "Make Makefile". It's a set of helpers included with Ruby that eliminates the finicky part of getting a C build set up. We call create_makefile and set a name for the extension. This creates a Makefile which contains all the configuration and commands to build the C code.

Next, we need to write some C code to connect the library to Ruby. We'll create some functions that convert C types such as char* to Ruby types such as String. Then we'll create a Ruby class with C code.

First off, we include some header files from Ruby. These will import the functions we need to do type conversion. We also include the library.h header file that we created earlier so that we can call our library.

#include "ruby/ruby.h"
#include "ruby/encoding.h"
#include "library.h"

We then create a function to wrap each function in our library. This is the one for string:

static VALUE string(VALUE self, VALUE value) {
  Check_Type(value, T_STRING);

  char* pointer_in = RSTRING_PTR(value);
  char* pointer_out = string_from_library(pointer_in);
  return rb_str_new2(pointer_out);
}

We first check if the Ruby value coming in is a string, since processing a non-string value might cause all sorts of bugs. We then convert the Ruby String to a char* with the RSTRING_PTR helper macro that Ruby provides. We can now call our C library. To convert the returned char*, we use the includes rb_str_new2 function. We'll add similar wrapping functions for number and boolean.

For numbers, we do something similar using the NUM2INT and INT2NUM helpers:

static VALUE number(VALUE self, VALUE value) {
  Check_Type(value, T_FIXNUM);

  int number_in = NUM2INT(value);
  int number_out = number_from_library(number_in);
  return INT2NUM(number_out);
}

The boolean version is also similar. Note that C doesn't actually have a boolean type. The convention is to instead use 0 and 1.

static VALUE boolean(VALUE self, VALUE value) {
  int boolean_in = RTEST(value);
  int boolean_out = boolean_from_library(boolean_in);
  if (boolean_out == 1) {
    return Qtrue;
  } else {
    return Qfalse;
  }
}

Finally, we can wire up everything so that we can call it from Ruby:

void Init_extension(void) {
  VALUE CFromRubyExample = rb_define_module("CFromRubyExample");
  VALUE NativeHelpers = rb_define_class_under(CFromRubyExample, "NativeHelpers", rb_cObject);

  rb_define_singleton_method(NativeHelpers, "string", string, 1);
  rb_define_singleton_method(NativeHelpers, "number", number, 1);
  rb_define_singleton_method(NativeHelpers, "boolean", boolean, 1);
}

Yes, you read that right: we can create Ruby modules, classes and methods in C. We set up our class here. We then add Ruby methods to the class. We have to provide the name of the Ruby method, the name of the C wrapper function that will be called and indicate the number of arguments.

After all that work, we can finally call our C code:

CFromRubyExample::NativeHelpers.string("a string")

Conclusion

We jumped through hoops, didn't crash and got our C extension to work. Writing C extensions is not for the faint of heart. Even when using the ffi gem you can still quite easily crash your Ruby process. But it is doable and can open up a world of performant and stable C libraries for you!

Ruby's redo, retry and next keywords

Thijs Cadier — Tue, 05 Jun 2018 15:12:30 +0000

We've talked about the retry keyword while discussing retrying after exceptions. Its little-known counterpart redo works similarly, but re-runs loop iterations instead of whole blocks.

The `redo` keyword

As we learned in the previous Academy article, retry lets you retry a piece of code in a block:

begin
  puts "Iteration"
  raise
rescue
  retry
end

This example prints the word "Iteration" to the console before raising an exception. The rescue blocks executes, which calls retry and starts the block again from the beginning. This results in our program endlessly printing Iteration. The redo keyword lets you achieve a similar effect when using loops. This is useful in situations where you need to retry while iterating for example.

10.times do |i|
  puts "Iteration #{i}"
  redo if i > 2
end

This will print:

$ ruby redo.rb
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 3
Iteration 3
...

Notice how the iteration count stays the same? It will step back execution to the start of the loop. This variant of the code using retry will print the exact same output:

10.times do |i|
  begin
    puts "Iteration #{i}"
    raise if i > 2
  rescue
    retry
  end
end

You can use redo to implement retrying in a loop. In the next example we have a queue of jobs. They either return :success or :failure. We keep re-running the same iteration of the loop until the job succeeds.

[job_1, job_2, job_3, job_4].each do |job|
  redo unless job.call == :success
end

Ruby 1.8

The behavior of retry and redo changed between Ruby 1.8 and 1.9. They used to restart the loop's iteration, but both in a different way. From 1.9, retry only works with a begin/rescue block and redo only works within loops.

The `next` keyword

If you want to move to the next iteration of the loop, as opposed to moving back to the start of the current one, you can use next.

10.times do |i|
  puts "Iteration #{i}"
  next if i > 2
  puts "Iteration done"
end

This will print:

$ ruby next.rb
Iteration 0
Iteration done
Iteration 1
Iteration done
Iteration 2
Iteration done
Iteration 3
Iteration 4
...

See how the iteration counter keeps incrementing? In most cases using next is what you want. Look at redo if you need a loop that runs an exact number of times or needs error handling when iterating over an array.

We hope you learned something new about redoing iterations in loops and would love to know what you thought of this article (or any of the other ones in the AppSignal Academy series). Please don't hesitate to let us know in the comments what you think, or if you have any Ruby subjects you'd like to learn more about.

DEV Community: Thijs Cadier

CPU Steal Time: A Crucial Metric for Cloud Servers and VMs

What Is CPU Steal?

CPU Steal in a Real-life Example

What to Do about CPU Steal Time

CPU Steal Time Is Low

CPU Steal Time Is Relatively High

How to Monitor CPU Steal with AppSignal

Low CPU Steal with Some Outliers

Consistently Higher CPU Steal

What Did We Learn?

Monitor Your Hosts and Get Stroopwafels

The Citadel Architecture at AppSignal

Monolith

Growing Pains

An Outpost

Life at the Citadel

Building a Ruby C Extension From Scratch

Our Library Ported to C

The 2018 Way: Use the ffi Gem

Wrapping our Library in a C Extension

Conclusion

Ruby's redo, retry and next keywords

The redo keyword

Ruby 1.8

The next keyword

The 2018 Way: Use the `ffi` Gem

The `redo` keyword

The `next` keyword