Debugging latency; serverless myths; & production secret management at Airbnb

#todayilearned #serverless #webperf

TL;DR notes from articles I read today.

Latency is a critical measure to determine whether our systems are running normally or not.
There are many collections libraries available that help you collect latency metrics.
Heat maps are useful as they help visualize latency distribution over time.
After narrowing down the source of the latency to a service or process, look at the host-specific and in-process reasons why latency occurred in the first place.
If the host is behaving normally and networking is not impacted, go and further analyze the in-process sources of latency.
Some language runtimes like Go allows us to internally trace runtime events in the lifetime of a request.

Full post here, 6 mins read

One myth is that serverless implies Functions as a Service (FaaS). Cloud services are serverless if no servers are exposed for you to administer, if they scale automatically and you pay for what you use only. In fact, serverless need not mean web-based apps, and can include real-time analytics and processing, so look beyond functions.
Don’t think that serverless is a silver bullet. Serverless technology is best suited for event-based architectures, rather than traditional client-server architecture, and you need to beware of recreating monolithic structures.
Another common myth is that serverless means an end to operational burdens. Advanced observability is intrinsic, so you need operational effort to monitor, maintain and effectively scale, though you need not administer servers.
Don’t believe that serverless is infinitely scalable. Serverless services have high availability but cannot scale infinitely - each service has limits, such as lambda’s memory limits and Kinesis’ throughput limits - so you need to optimize for the limits and plan for failure scenarios to ensure resilience.

Full post here, 6 mins read

Airbnb built an internal tool Bagpiper which is a collection of tools and framework components that it uses for the management of production secret assets.
They designed it to decouple secret management from other app configurations as Airbnb scaled, and to ensure a least-privileged access pattern, encryption of secrets at rest, support for applications across several languages and environments, and managing secrets for periodic rotation.
Bagpiper creates segmented access by asymmetrically encrypting secrets with service-specific keys: a secret is encrypted with each of the public keys on a per-secret keychain, and only services with the corresponding private keys can decrypt the secret. It encrypts information at rest and decrypts it during use.
Engineers can add, remove and rotate secrets, and make them available to select production systems. Secrets and changes to code are typically deployed together.
Secrets are rotated continuously, using secret annotations that specify when a secret was created/last rotated and when to rotate it again.

Full post here, 6 mins read