DEV Community: Amberflo.io

Best practices for metering cloud resources

Amberflo.io — Fri, 23 Sep 2022 15:21:59 +0000

Originally posted on www.amberflo.io/blog on August 1, 2022.

Metering sounds simple enough at first glance, but upon taking a closer look, it becomes clear why companies like AWS, Snowflake, and Twilio have dedicated entire engineering teams to the construction and maintenance of their internal usage metering systems. As we have written before, metering is a heavy lift, and there is very little margin for error in getting it wrong. One of the key design principles that we have upheld since day one is building a flexible, scalable, and domain-agnostic metering solution for the world. To accomplish this, our Metering Cloud must be able to accurately track and aggregate any resource, infrastructure, or user action possible.

One of the critical aspects of this is handling different types of usage events and reporting methodologies.

‍
Cloud Resources (single vs long-lasting)
Amberflo allows you to meter both instantaneous, discrete events and long-lasting events. Long-lasting events describe non-momentary resource consumption at a certain frequency or scale (known as the usage rate).

Single and Long-lasting Events
Single events plotted over time can be thought of as points in the mathematical sense, having no length since they do not take place over time, they simply take place. As an example, consider you want to meter the count of API calls for some endpoint. The endpoint sends a meter event to Amberflo for each completed API call indicating that it took place. These events can be plotted as points over time (based on the timestamp received at ingestion), and then counted up when queried to return the number of calls handled over a given time.

On the other hand, long-lasting events do take place over time, so when graphed over time they can be represented as curves with nonzero length. It is important to note that for long-lasting events, the rate of usage may change throughout the time that an event is taking place. For example, consider a data storage solution; from the moment the first dataset is ingested for storage lasting until the last data is removed from the system, resources are being consumed. That said, the rate of consumption changes as data is ingested and extracted from the system and the amount of storage in use changes.

‍
Reporting Single Events
Relative to long-lasting events, reporting single events is relatively straightforward.

In Amberflo, simply define the meter, associated dimensions (if any), and send in the count as the value of the meter. Select Sum as the meter aggregation type, and Amberflo automatically ingests, persists, aggregates the events, slices the aggregate values over time-series and presents the data back to you via dashboard and API in real-time.

Reporting Long-lasting Events
Reporting for long-lasting events presents one unique challenge that isn’t present for single instantaneous events, and that is correctly reporting stop events. In the instantaneous case, if a record is not sent to Amberflo, then the consequence is simply that the one single record is missing, causing the total count to be incorrect by whatever the missing value for that event is. In the long-lasting case, if a start event for a resource is received but for some reason the stop event fails to arrive, then the meter will be tracked as if that usage goes on forever (until a stop event is received). If you are billing the customer for that usage, clearly this has the potential to cause massive problems very quickly.

To address this, we provide a timeout period. If a start event is received and no stop event is received within that timeout period, then the meter is automatically reset as if a stop event were received. We allow you to set a timeout on two levels: globally at the meter definition level (the default is set to one year), and at the usage event level by setting “aflo.expiration_time_seconds” equal to the max anticipated (or allowed) usage for that meter event.

To have further built-in resilience, we recommend employing a “heartbeat” approach to reporting usage for long-lasting events when employing the momentary reporting approach (see below). That is, define some small time interval within the timeout period where you will constantly report the current usage (or value) at that time. For example, if you have set a timeout period of 30 minutes, then you may use heartbeat intervals of 5 minutes; every 5 minutes, the system would send an event to Amberflo with the current usage rate, even if it hadn’t changed from the previous reporting. This way, if a stop event is missed for any reason, and no subsequent heartbeat events are received, Amberflo will automatically mark the end of the long-lasting stage. This allows you to create an upper bound for maximum possible time that you could record incorrect usage for, based on the length of the heartbeat intervals. If you want to keep the threshold lower, set a lower limit for timeout and higher frequency of heartbeats within the timeout.
‍

Momentary Reporting Method
As a best practice we recommend a momentary reporting method, that is where the account sends the value of the usage rate for the resource at the moment of reporting. Using the same storage example from above, the momentary usage events sent to Amberflo would be:

‍
{time: 9:00am, value: 8}
{time: 11:00am, value: 11}
{time: 11:30am, value: 7}
{time: 11:50am, value: 0}
‍

Using the meters from the example above, we can demonstrate the momentary reporting method with added heartbeat meters sent every 30 minutes (shown in yellow) as follows:

‍
{time: 9:00am, value: 8}
{time: 9:30am, value: 8}
{time: 10:00am, value: 8}
{time: 10:30am, value: 8}
{time: 11:00am, value: 11}
{time: 11:30am, value: 7}
{time: 11:50am, value: 0}
‍

You can see in this example how having the heartbeat meters allows for the possibility of losing a meter event and still metering accurately. If event 1 is dropped and never sent to Amberflo, without the heartbeat, the customer consumes 8 of the metered units which will not be tracked. Essentially, the customer would receive the 8 free units over 2 hours for free. With the heartbeat events, if event 1 is missed, only 30 minutes of free usage would be allowed for the customers, since at 9:30am a heartbeat meter with value 8 would be sent, correcting the status. If any of the heartbeat events are missed over the 2 hours between 9:00am and 11:00am, it will not cause errors since the other heartbeat meters provide built-in contingency.

As another example, we can consider event 7. Let’s first consider the case where only start, stop, and updated value meters are used (no timeout set or heartbeat meters sent). If that event is missed and not reported to Amberflo, then with no stop event is incoming, the customer will be billed as if they consumed those 7 metered units forever.

By employing a timeout period, the customer cannot be erroneously billed ‘forever’ if a stop event is never received. Whatever the length of the timeout period is the max amount of time that a customer can be billed for erroneously if a stop event is not sent. In the above example, let’s suppose we define a timeout period of 3 hours; then in the worst case scenario, event 7 is missed and no stop event is forthcoming. Then at 2:30pm (3 hours after the most recent meter sent), the meter value would be reset to 0 as if a stop event were received.

A best practice would be to employ both, a timeout period and the use of heartbeat meters. This will minimize the impact of missed meter events and allows you to set an upper bound for the potential reporting error of the system. In the above example, with the 3 hour timeout period and 30 minute heartbeat intervals, if event 7 is missed, a heartbeat event would be incoming 30 minutes after the most recent event (event 6) at 12:00pm with a value 0. If that event were to be missed, additional meters would be coming every 30 minutes until the timeout at 2:30, so there would be several chances to correct the missed event, and it would certainly result in the lowest error of all the examples we discussed.

‍
Best practices
To tie it all together, we have three key best practices for reporting long-lasting events.

Wherever possible employ the momentary reporting method with regular heartbeat events. Send an event with the momentary value whenever the usage rate is changed, and send heartbeat events every X minutes (defined by the timeout period) to minimize inaccuracy in case of a missed event.
Set a tight global timeout on the meter level. Consider the maximum amount of time that each resource you are metering might realistically be consumed for. This timeout eliminates the possibility of missing a stop event and erroneously recording a long-lasting event that never stops.
Add a timeout at the event level wherever possible. For each event, it may be possible to calculate a more precise timeout value; for example, you might correlate the event-level timeout for a storage solution to the amount of data being ingested. Larger ingests might have longer timeouts while smaller files would time out more quickly. By monitoring your system over time you can calculate these event-level values for your own resources and usage patterns.

Meter ingestion options for high throughput metering use cases

Amberflo.io — Tue, 13 Sep 2022 16:07:28 +0000

Originally posted to www.amberflo.io/blog on 8/30

Amberflo was built with the highest throughput use cases in mind – think billions of events per week – to accomplish this, we employed a serverless architecture that auto-scales to ensure consistent performance regardless of load. Still, for some use cases where the event volume reaches into the billions per customer (such as messaging or API monetization), the generic event-based payment model becomes unfeasible, even with volume-based discounting in place.
To ingest events, Amberflo offers a variety of options depending on the customer’s infrastructure and organizational preferences. The primary method that we recommend is using our SDKs, as these can be integrated closest to the source (where events are being generated in the client system) and can deliver the lowest possible latency. We also allow customers to connect directly to our API endpoints without using SDKs, in case their backend language is not supported or if they prefer to interact with cloud services such as S3 directly. Beyond those methods, Amberflo can take meter data from a CSV file, from a third-party logging or monitoring solution, or from a cloud storage solution like S3. If a customer has an existing meter data pipeline, we can work with them to direct the events to Amberflo.
Amberflo SDKs are built to operate asynchronously with automatic batching and flushing; this means that when an event is generated by a client system and sent to the Amberflo SDK, it is not immediately sent back to Amberflo for aggregation but instead, batches are sent at preconfigured time intervals (batch size and send interval are configurable). After the batch is sent, the cache is flushed and new events can be received.
‍
Aggregate and Squash at the Source (Non-Distributed)

For use cases demanding ultra high volume metering such as large-scale messaging or API monetization, the standard ingestion method using our SDKs may not be feasible. In that case, despite the batching, each event is received and aggregated by Amberflo on a one-to-one basis. Even with volume-based discounting, this can quickly become cost-prohibitive using this approach. Instead, we recommend squashing these batches by aggregating some of these events at the source before ingestion to Amberflo.
For example, consider a batch of 1000 events (or 10 seconds’ worth with a standard batch size of 100 per second). This batch can be squashed at the source by aggregating the similar meters (similar meaning meters associated with the same customer, and having the same dimensions) and reporting the total aggregate value to Amberflo. Suppose in that batch of 1000 events (representing API calls), only 5 customers accessed the API; then what could be reported to Amberflo would be ingested as 5 events, one for each customer, corresponding to number of calls each customer made to the API over that 10 seconds (with the total across all 5 customers adding up to 1000).
‍
Pre-Aggregation in a Distributed Context

In a distributed system, an additional step needs to be made of unifying and staging the data in an intermediate location such as S3. In a distributed system, meter events are generated at multiple different sources (such as multiple different servers or IoT devices). There needs to be a way to unify all the events that occurred over a given time period, regardless of where they originated. We recommend staging all events in S3 before aggregating or ingesting to Amberflo. After joining the data across locations, similar events can be aggregated as described above, with the aggregate totals reported to Amberflo at configurable intervals.
In this case, consider a factory with four different assets (machines) which complete 100 operations per second; every 10 seconds, batches of events from Assets A, B, C, and D are sent to S3 indicating success or failure of the machine for each operation. There, these events are squashed further by aggregating all of the successes and failures, then reporting these totals to Amberflo as two separate events (success total and failure total), rather than reporting the individual successes and failures to Amberflo.
‍
Controlling Reporting Granularity

The trade-off when employing this approach is a slight loss in granularity. Since you are reporting aggregated values to Amberflo, there is no way to recover the specific values if they are needed at a later date. Continuing the above example, you would be able to see the failure rate across assets A-D (updated every 10 seconds), but you would not be able to see how each asset individually was performing using this approach. That said, by configuring the batch sizes and reporting interval, you have some control over the level of granularity being generated.