DEV Community: Michael O'Brien

IoT Security Updates

Michael O'Brien — Thu, 17 Jul 2025 03:24:55 +0000

Many companies have demonstrated the value of regularly enhancing product performance through software and firmware updates. Apple is a well-known example, delivering seamless device updates that improve functionality and user experience. But they're not alone—businesses of all sizes are increasingly using remote updates to boost product capabilities and address security concerns.

In fact, it's becoming a legal obligation in many regions to provide security updates throughout a device’s lifetime.

The European Union has introduced the Cyber Resilience Act (CRA), a regulation aimed at enhancing cybersecurity for IoT products. This legislation mandates that manufacturers ensure their products are secure throughout their entire lifecycle, from design to decommissioning. This requires that software updates are provided for the lifetime of the device.

That said, smoothly updating a fleet of devices—without issues or downtime—can be a real challenge.

With the Ioto Update Manager, you can create, deploy, manage, and monitor over-the-air updates for your IoT devices, ensuring they remain secure, functional, and up-to-date.

Ioto Iot Device Update

The Ioto IoT Update Manager offers the following capabilities:

Upload device software images for dissemination
Distribute software images efficiently and reliably using a global CDN
Use distribution policies to target specific subsets of device populations
Implement gradual update rollouts
Throttle update rate to minimize load and risk
Ensure secure and dependable software delivery through TLS and cryptographic checksums
Track customer base update progress with detailed reports and metrics

The Ioto update manager allows you to selectively distribute updates to any chosen group of devices based on a distribution policy. This enables you to update all your devices or only specific groups as needed.

Components of Device Update

The Ioto update solution has three major components:

Item	Description
Builder	Portal to create and manage software updates and distribution policies
Ioto Cloud Service	Service to securely store software updates and distribute via a global CDN
Device Agents	Device-resident software to poll, download and apply software updates to the device

The Builder assists in preparing a software update for distribution, which includes uploading the device software image, specifying the version, and setting a distribution policy. It also offers comprehensive monitoring and reporting of your device population and update performance.

The Ioto Cloud Service hosts the device software images and facilitates communication with devices to deliver the updates to the relevant devices.

The Device Agents contain the necessary logic to interact with the cloud service, enabling them to download and install new software images as they become available.

Device Agent Support

Using the Builder, you can deploy updates to any device, regardless of the device agent you utilize. The Ioto device agent pre-integrates the software update capability. Other device agents can use the stand-alone EmbedThis Updater. The GoAhead and Appweb device agents bundle the Updater code with their release distributions.

The EmbedThis Updater is a command line utility that can request, fetch and apply software updates. Versions are provided in three forms:

C program utility
C program library
NodeJS utility

For example:

updater -v \
    --device "ABCDEF1234" \
    --host "https://abcdefghij.execute-api.ap-southeast-1.amazonaws.com" \
    --product "000001234567890AAKW996CZHH" \
    --token "00000001234567890AABBEGYJB" \
    --version "1.2.3" \
    --file updater.bin \
    --cmd ./apply.sh \
    model=pro ports=16 memory=256

Regardless of the device agent you use, the underlying update API and Builder Update service is the same.

Creating Device Updates

To define a device software update, you supply the following parameters to the Builder:

Target Device Product ID
Software update version number
Software Update image
Distribution policy
Update distribution limits
Rollout pacing factors
Distribution Device Cloud
Update description

Product Selection

A Builder account may manage many devices that have different update policies and usually require different firmware. When defining a software update, you nominate a Builder product definition for which the update applies.

For each family of devices that require the same software update images, you should create a Builder product definition. When the product definition is created, the Builder also creates a product ID token. This token is included in the device upgrade request to select the appropriate product and software update. The product token is obtained from the Builder Token List after creating the product definition.

The Builder uses the product ID token paired with the update distribution policy to define the subset of devices that are eligible to receive the update. To receive a software update, a device specifies a Builder Product Token that selects the product for which software updates may be published.

Device Cloud Selection

Software updates are reliably stored in a device cloud and distributed globally via the AWS CDN to local regions. When defining updates, you can select your device cloud to store and manage the updates. Select the device cloud from the pulldown list. The update facility is designed to scale and will handle device populations up to and beyond 10,000,000 devices.

Software Version

The software update version number is your device's version number. The version numbers must be compatible with the SemVer 2.0 version specification.

For Ioto, the current version for your device is defined via the version property in the ioto.json5 configuration file. For other device agents, the version is provided in the update API request.

The update description can be an informative description for your purposes. It is recommended to describe the purpose and extent of the update.

Software Distribution

At regular intervals, and typically once per day, device agents should connect to the Device Cloud for a "checkin" to see if any update has been published. During the checkin, the device agent will submit the Product ID, Device ID and other device-specific information that can be used when evaluating the distribution policy to determine if an update is available and suitable for this device.

If you are running the Ioto agent, it will automatically perform a checkin according to the schedule defined in the ioto.json5 configuration file. If you are using the EmbedThis Update utility, you should schedule that to run regularly using Cron or a similar facility.

The device cloud service will evaluate the distribution policy expression when the device checks-in. The device cloud will retrieve the most recent updates and check the updates in reverse version order and select the first matching update for the device. If the policy matches for the device, the URL for the update image will be returned to the device agent.

The device agent will then download the update image and verify the integrity of the update image. If verified, an update script is invoked to apply the update.

If you are running the Ioto agent, the "scripts/update" script will be invoked to apply the update. You should customize this script to suit your device. If you are running Ioto on an RTOS, without scripting, you will need to watch and react to the Ioto event device:update using the rWatch API.

If you are running the EmbedThis updater, you should customize the apply.sh script to apply the update to your device.

Software Distribution Policy

The update distribution policy enables you to target specific relevant subsets of your device populations.

The distribution policy is a simple JavaScript-like expression that is evaluated by the device cloud at runtime to determine if the update is relevant for a specific device. If you leave the policy blank, then all devices with a version that is earlier than the software update version will be updated.

Here is a sample policy expression:

major >= 1 && minor >= 1 && patch >= 5 && memory >= 256 && ports == 32

The device properties submitted with the update request are made available as expression variables. In this case, the "memory" and "ports" variables are supplied with the update request. If you are using the Ioto device agent, these variable can be defined in the device.json5 file.

The full device version is accessible as the policy variable "version" and the version of the software update is provided via the "newVersion" variable. The device version string is also split into SemVer components: major, minor and patch.

The default policy is uses the inbuilt function semver which compares two version strings. This default policy compares if the current version is earlier than the new update version using the following expression.

semver(version, "<", newVersion)

The policy expression language understands the types: Numbers, Boolean, String literals, Regular Expressions and null. Strings are quoted with either single or double quotes.

Sub-expressions can be grouped with parenthesis and the boolean operators && and || can group conditional operands. Regular expressions (delimited by slashes) may be used with the "==" and "!=" operators. The regular expression can be on either side of the operator.

The policy expression is run-time limited to evaluate up to 50 expression terms. This is to protect the device cloud and service against denial of service attacks.

Update Rollout Limits

Implementing a gradual update strategy can help minimize load and risk. Updating a large number of devices simultaneously can impose an excessive burden on your service, so staggering the rollout can distribute the load more evenly.

Despite thorough testing, some updates might still be considered "risky." To minimize this risk, you can update a small subset of your device population first to see if the update causes any issues.

The update service provides update limits and gradual rollout factors that allow you to control the rate of updates. The following mechanisms are supported:

Max device limit and device percentage limits define the maximum number of devices that can be updated. Once either limit is reached, further updates are suspended.
For gradual rollouts, you can define an update rate. This is implemented via a max updates per period, which limits the number of updates to a specified number of updates over a defined time period. For example, you could set a limit of 1000 updates per hour (3600 seconds).

If a device meets the distribution policy and rollout limits, the URL for the update image will be returned to the update agent.

Analytics, Metrics & Reports

The Builder provides extensive analytics to track the progress and performance of updates.

The Ioto device cloud tracks metrics uniquely per-product and for each product update version.

Metric	Dimensions	Description
UpdateTotal	Product, Product/Version	Total number of devices
UpdateDeferred	Product, Product/Version	Number of updates temporarily deferred due to rollout policies
UpdateStarted	Product, Product/Version	Number of updates started
UpdateSuccess	Product, Product/Version	Number of successful updaes
UpdateFailed	Product, Product/Version	Number of failed updates

The Builder Update list also includes metrics for tracking how many devices are using each update:

Summary

With the Ioto IoT Device Updater, you can seamlessly update your devices to quickly and reliably address security issues and deliver increased functionality and performance to your users.

Want More Now?

To learn more about EmbedThis Ioto, please read:

IoT AI with Ioto

Michael O'Brien — Thu, 17 Jul 2025 03:08:46 +0000

Artificial Intelligence (AI) significantly enhances edge devices by enabling more intelligent, autonomous operations. The recent advances in large language models (LLMs) running in the cloud are leading to transformative applications in the IoT space.

Developers typically select from three principal AI integration patterns: on-device models, cloud-based models, and hybrid models.

On-device language models operate entirely within the local hardware environment. This approach offers data privacy, reduced latency, and consistent operation regardless of network conditions, making it ideal for real-time applications or devices with intermittent connectivity or stringent privacy requirements. However, the complexity and scale of these models are constrained by the limited computational resources of edge devices.

Cloud-based language models offload computationally intensive processing to cloud servers, enabling the use of robust, large-scale LLMs that surpass the resource capabilities of edge devices. This design provides advanced features, seamless scalability, and simplified updates. Nevertheless, it relies on continuous internet connectivity and may introduce latency.

Hybrid approaches combine on-device models with cloud-based models. In this pattern, tasks that are privacy-sensitive or critical are executed locally, while more complex, resource-intensive operations that are not time-sensitive are processed in the cloud. This approach effectively blends the strengths of privacy and responsiveness with the new capabilities offered by the latest generation of cloud-based models.

Ioto IoT AI

Ioto provides an intuitive AI library that simplifies interactions with cloud-based LLMs, facilitating tasks such as data classification, sensor data interpretation, information extraction, and logical reasoning. This capability is particularly beneficial for applications in predictive maintenance, smart agriculture, healthcare, smart homes, and environmental monitoring. It’s ideal for analyzing non-real-time sensor data using the power of an LLM.

The Ioto AI library can invoke cloud-based LLMs and run local agents that operate with full access to device context.

OpenAI and Foundation Models

Ioto supports the standard Chat Completions API and also implements the newer OpenAI Response API which is designed to help developers create advanced AI agents and workflows capable of performing cloud-based tasks like web searches, file retrievals, and invoking local agents and tools.

The standard OpenAI Chat Completions API is supported by most other foundation models. We expect other vendors to follow OpenAI's lead and add support for the Response API to their offerings.

Getting Started with Ioto AI

The Ioto distribution includes an ai sample app that demonstrates the AI facilities of Ioto.

Before building, you need an OpenAI account and an API key. You can get an API key from the OpenAI API Keys page.

Once you have an API key, you can edit the apps/ai/config/ioto.json5 file to define your access key and preferred model. Alternatively, you can provide your OpenAI key via the OPENAI_API_KEY environment variable.

ai: {
    enable: true,
    provider: "openai",
    model: "gpt-4o",
    endpoint: "https://api.openai.com/v1",
    key: "sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
services: {
    ai: true,
    ...
},

The services.ai controls whether the AI service is compiled, while the ai.enable setting enables or disables it at runtime.

If you are using another foundation LLM other than OpenAI, you can define the API endpoint for that service via the endpoint property.

Building Ioto and the AI Sample App

When you build the Ioto Agent, you can select the ai sample app to play with the AI capabilities. This will enable the Ioto AI service.

make APP=ai

AI Sample App

The AI sample app has five web pages that are used to initiate different tests:

App	Page	Use Case	API Case
Chat	chat.html	ChatBot	Use the OpenAI Chat Completions API
Chat	responses.html	ChatBot	Use the new OpenAI Response API
Chat	stream.html	ChatBot	Use the new OpenAI Response API with streaming
Chat	realtime.html	ChatBot	Use the OpenAI Chat Real-Time API
Patient	patient.html	Patient Monitoring	Use the new OpenAI Response API and invoke sub-agents

The Patient app demonstrates a patient monitor using an AI agentic workflow. The app measures a patient's temperature locally by calling the getTemp() function. It sends the temperature to the cloud LLM which determines if the patient is in urgent need of medical attention. If so, it responds to instruct the device workflow to call the ambulance by using the local callEmergency() function. The web page has two buttons to start and stop the monitoring process. This app demonstrates the use of the OpenAI Response API and local agent functions.

The Chat demo is similar to the consumer ChatGPT website. Each web page is a simple ChatBot that issues requests via the Ioto local web server to the relevant OpenAI API. The requests are relayed to the OpenAI service and the responses are passed back to the web page to display.

The sample apps register web request action handlers in the aiApp.c file. These handlers respond to the web requests and in turn issue API calls to the OpenAI service. Responses are then passed back to the web page to display.

Note: The AI App does not require cloud-based management to be enabled.

Code Example

Here is an example calling the Responses API to ask a simple question. This example uses file search (aka RAG) to augment the pre-trained knowledge of the LLM and a local function to get the local weather temperature.

#include "ioto.h"

void example(void)
{
    cchar *vectorId = "PUT_YOUR_VECTOR_ID_HERE";
    char  buf[1024];

    /*
        SDEF is used to concatenate literal strings into a single string.
        SFMT is used to format strings with variables.
        jsonParse converts the string into a JSON object.
     */
    Json *request = jsonParse(SFMT(buf, SDEF({
        model: 'gpt-4o-mini',
        input: 'What is the capital of the moon?',
        tools: [{
            type: 'file_search',
            vector_store_ids: ['%s'],
        }, {
            type: 'function',
            name: 'getWeatherTemperature',
            description: 'Get the local weather temperature',
        }],
    }), vectorId), 0);

    Json *response = openaiResponses(request, agentCallback, 0);

    // Extract the LLM response text from the json payload
    text = jsonGet(response, "output_text", 0);
    printf("Response: %s\n", text);

    jsonFree(request);
    jsonFree(response);
}

The openaiResponses API takes a JSON object that represents the OpenAI Responses API parameters. The SDEF macro is a convenience to make it easier to define JSON objects in C code. The SFMT macro expands printf-style expressions. The jsonParse API parses the supplied string and returns an Ioto Json object which is passed to the openaiResponses API.

The response returned by openaiResponses is a JSON object that can be queried using the Ioto JSON library jsonGet API. The output_text field contains the complete response output text.

The agentCallback function is triggered when the LLM needs to invoke a local tool. It is defined as follows:

static char *agentCallback(cchar *name, Json *request, Json *response, void *arg)
{
    if (smatch(name, "getWeatherTemperature")) {
        return getTemp();
    }
    return sclone("Unknown function, cannot comply with request.");
}

Consult the Responses API for parameter details.

Want More?

See the apps/src/ai app included in the Ioto Agent source download for the example responses.html web page that uses the Responses API and the patient.html web page that uses the OpenAI Response API and local agent functions.

Here is the documentation for the Ioto AI APIs:

Consult the OpenAI documentation for API details:

Reach out if you have any questions or feedback by posting a comment below or contacting us at sales@embedthis.com.

Wrapping Up

Ioto makes it easy to integrate powerful AI capabilities into your IoT devices—without the complexity. Whether you're building a smart appliance, medical sensor, or environmental monitor, Ioto’s flexible architecture and built-in AI tools help you move faster.

Download the Ioto Agent, try out the sample apps, and start building smarter devices today.

The Future of IoT AI in 2025 and Beyond

Michael O'Brien — Thu, 17 Jul 2025 03:03:21 +0000

Machine learning (ML) has become a cornerstone of smart, autonomous decision-making in IoT devices. These “smart devices” derive their intelligence from the ability to analyze and act quickly on sensor data, at the edge, and respond accordingly.

Historically, microcontrollers were too limited for anything beyond basic rule-based logic. But with the advent of frameworks like TensorFlow Lite for Microcontrollers, we entered the era of TinyML, enabling machine learning on even the most resource-constrained devices.

Edge AI Meets Cloud: The Rise of Hybrid IoT AI

While device-based models have steadily benefited from better microcontrollers and model optimization techniques, the AI landscape has seen an explosive leap in cloud model capabilities in the past year. Foundation models such as OpenAI’s GPT-4, Anthropic’s Claude, and Google’s Gemini are now so advanced they can understand, generate, and reason across multiple modalities—text, speech, image, and sensor data.

These models are now capable of powering use cases that were literally unthinkable just a couple of years ago. As these cloud models continue to evolve, the boundary between edge and cloud capabilities is rapidly shifting.

This changes the game for edge AI. Rather than a simple migration from cloud to edge, we’re now seeing a hybrid IoT AI architecture emerge. Edge devices handle real-time, low-power inference, while the cloud provides deep reasoning, personalization, and large-scale pattern recognition.

Edge AI is still essential — especially for real-time, low-latency, or privacy-focused applications. But as cloud AI grows exponentially in power and flexibility, hybrid architectures that combine edge inference with cloud intelligence are becoming the new standard.

IoT AI no longer means all inference happens on the device.

A modern IoT device might use edge ML to locally detect a wake word, capture sensor anomalies, or manage immediate control loops — and then stream relevant data to a cloud model for deeper analysis, anomaly detection, or personalized insights. This hybrid model unlocks the best of both worlds: instant local reactions and the nearly limitless compute of the cloud.

This paradigm shift doesn’t mean edge ML is obsolete — far from it. But the growing capability of cloud AI means edge models don’t need to carry the full burden of intelligence. Instead, edge devices are evolving into smart front-ends that delegate deeper reasoning and processing to the cloud.

New Possibilities

This unlocks new possibilities for IoT AI applications:

On-demand intelligence: Devices can dynamically invoke cloud AI only when needed — for example, to send a low-res image for anomaly classification or trigger a cloud-based automation workflow.
Context-aware edge devices: A smart home assistant can locally detect movement, then query a cloud model to determine whether it’s a pet, an intruder, or a family member—using household context and historical behavior.
Edge-tuned cloud services: Platforms like EmbedThis Ioto offer APIs that let devices feed structured sensor data to cloud models, such as custom GPTs fine-tuned for your factory floor.

Cloud models can do things today we couldn't imagine last year

IoT Platforms like EmbedThis Ioto are now integrating direct IoT AI APIs to cloud models alongside the ability to run local inference.

Why Run ML on Microcontrollers?

Machine learning is still important for IoT devices. Everyday electronics can become “smarter” by directly integrating ML models into microcontrollers. This means they can function without relying on an external processor or constant cloud access for tasks like signal processing, speech recognition, scene recognition, predictive maintenance, or anomaly detection.

Running ML models directly on microcontrollers enables:

Real-time processing with minimal latency
Energy efficiency, vital for battery-powered or low-power devices
Improved privacy and security by keeping data local
Device Autonomy for remote, disconnected or bandwidth-constrained environments

We must now consider what AI tasks should run locally and what should run in the cloud.

As microcontrollers continue to improve in capability — with better DSPs, more onboard RAM, and built-in AI accelerators — they can support increasingly sophisticated models. However, developers must now think not just about what can run locally, but what should run locally versus in the cloud. For instance, basic anomaly detection may happen at the edge, while complex root-cause analysis is handled in the cloud by large foundation models.

The Role of Cloud Models

Despite the rapid evolution of edge hardware, some AI tasks are too large, complex, or data-hungry to run efficiently on embedded hardware. While edge devices excel at fast, local decision-making, there’s a growing class of applications that benefit from offloading high-level inference and reasoning to the cloud.

Thanks to accessible cloud-based AI APIs and low-latency connectivity, edge devices can invoke powerful foundation models on demand—tapping into capabilities like natural language processing, multimodal and deep reasoning.

This enables a flexible, dynamic collaboration between edge and cloud. For example:

A low-powered environmental sensor can summarize temperature trends locally, then call a cloud model to predict equipment failure based on similar historical patterns.
A logistics scanner might capture visual damage indicators and request cloud-based assistance to classify the severity and recommend next steps.
A medical wearable can track biometric data in real time, but push anomalies to a cloud model trained on population-scale datasets for further analysis and multilingual patient feedback.

These scenarios are no longer experimental. Enterprises and device makers are deploying them today—using model APIs from a growing ecosystem of providers. Whether leveraging large language models, vision transformers, or speech models, the goal is the same: push only what’s needed to the cloud, and only when it adds value.

Even on the device itself, we're beginning to see tiny variants of large models—quantized, distilled, or pruned—running directly on AI-capable microcontrollers and edge SoCs. This allows for partial inference locally, followed by cloud-based reasoning. For example:

A compact vision model might detect movement on-device, while a cloud model interprets the activity to identify species under threat.
A quantized NLP model might perform wake-word detection or intent classification at the edge, with complex dialogue managed in the cloud.

The takeaway? Cloud-based large models aren’t replacing edge ML—they’re augmenting it. As hardware and software evolve, we’re moving toward a more nuanced AI stack where tasks are dynamically split between device and cloud depending on compute needs, latency requirements, and context.

When to Use Edge vs. Cloud AI

Item	Edge	Cloud
Latency	Ultra-low (ms)	Higher, network-dependent
Privacy	High – data kept local	Depends on cloud platform
Model Size	Tiny (KB–MB)	Massive (GBs–TBs)
Scalability	Limited by device resources	Unlimited by cloud resources
Power Usage	Low	High
Use Cases	Real-time reaction, offline ops	Contextual reasoning, NLP, Pattern recognition
Best For	Safety-critical, mobile, wearable	Deep analytics, multimodal tasks

The future is not about choosing edge or cloud—it’s about orchestrating the best of both.

How to Invoke Cloud Models

Edge devices can invoke cloud AI in multiple ways:

Direct Call: Use REST API or WebSocket to call the cloud model and get a response.
Agentic Workflow: The device agent provides a suite of tools (functions) that can be invoked by the cloud model (indirectly) as responses are received and processed by the device.
Automated Trigger: Device sensor data posted by the device to the cloud triggers IoT platform automations that then invoke the cloud model with the device data and return the result to the device.

What IoT AI Really Means for 2025

While machine learning on microcontrollers has become more capable, the real story of 2025 is the rise of collaborative intelligence between edge and cloud. Edge ML frameworks like TensorFlow Lite continue to evolve — but they now sit within a broader AI ecosystem powered by powerful cloud models.

For developers, this means new design choices and new tradeoffs. You no longer need to cram every ounce of intelligence into a tiny microcontroller. Instead, you can architect your system to act fast at the edge, think deep in the cloud, and unlock IoT AI experiences previously out of reach.

EmbedThis Ioto

EmbedThis Ioto is a modern IoT Meta-platform designed to simplify the deployment of IoT AI-powered, connected devices. At its core is a compact, high-performance device agent that bridges edge intelligence with cloud-scale AI—enabling smart devices to both run local machine learning models and invoke powerful foundation models via direct cloud APIs.

With Ioto, edge devices can:

Run TinyML inference locally in parallel with other device operations.
Call cloud-based foundation models for tasks that require deeper reasoning, analysis, large language processing, or multimodal understanding.
Run agentic workflows locally using local agents and cloud-based models triggered by device data events or cloud-based automations.
Seamlessly sync data and state between the device and cloud with built-in MQTT, WebSockets, and RESTful HTTP support and then trigger cloud-based models and workflows based on device data events.

Cloud Model Integration

Ioto supports direct access to foundation model APIs, enabling devices to send structured data (e.g., sensor readings, text prompts, command requests) to the cloud and receive rich, contextual responses.

Ioto also supports invoking cloud models via automated triggers that monitor device data in the cloud and invoke models to analyze the data and generate responses and run workflows.

Ioto supports the following APIs:

OpenAI Chat Completions API – for conversational AI and natural language processing
OpenAI Response API – for generating structured outputs and invoking agentic workflows
OpenAI Streaming API – for real-time interaction with minimal latency
OpenAI Real-Time API – for continuous input/output workflows such as live monitoring or control

Although the default integration targets OpenAI, the API design is model-agnostic and compatible with any provider that supports the Chat Completions-style interface—including models from Anthropic, Mistral, Google, and open-source deployments using tools like OpenRouter or OpenLLM. It is anticipated that many other cloud providers will add support for the newer Responses API in the future.

Lightweight but Fully Equipped

Despite its small footprint—less than 300K of code—Ioto packs a comprehensive feature set:

Embedded HTTP web server with TLS for secure local UIs and APIs.
MQTT client and HTTP client for robust cloud connectivity.
Built-in WebSockets support for real-time bi-directional communication.
Embedded database and JSON parser for structured local data handling.
Over-the-air (OTA) firmware updates.
Tight integration with AWS services, including secure identity, storage, and messaging.
Cloud LLM Integration for OpenAI, Anthropic, Mistral, and Google.

Together, these capabilities make Ioto a versatile platform for building hybrid IoT AI architectures. Developers can deploy lightweight models directly on-device for speed and privacy, while calling on large cloud models for deeper, contextual tasks—without needing to reinvent their stack or overburden their microcontroller.

Whether you're building smart appliances, industrial sensors, or edge gateways, Ioto offers a future-proof foundation for IoT AI-enabled devices that think fast locally and think big in the cloud.

References

Consult the OpenAI documentation for API details:

Why 75% of IoT Projects Still Fail – and How to Beat the Odds

Michael O'Brien — Wed, 16 Jul 2025 07:36:15 +0000

The Internet of Things (IoT) is finally delivering on its promise. As of 2025, 85% of organizations are running IoT projects, and 88% consider IoT critical to their business success¹. With global IoT spending heading toward $1 trillion², enthusiasm is high. But success isn’t guaranteed. Many projects stall or collapse before reaching production.

Only 1 in 4 projects are deemed to be successful.

TL;DR:
Despite rising adoption, most IoT projects still fail due to unclear goals, poor integration, and scalability issues. This post outlines the latest failure stats and what businesses can do to improve outcomes.

Latest IoT Project Statistics

Recent industry surveys reveal that the majority of IoT projects do not achieve their intended outcomes. Estimates of IoT project failure rates typically range from 60% up to 80%. In other words, only roughly 1 in 4 IoT initiatives is ultimately considered successful.³⁴

High Failure Rates: A 2024 analysis notes: "surveys consistently find that 80% of IoT projects don’t reach successful deployment."⁴ Likewise, IoT experts estimate around 75% of IoT projects fail to achieve their desired results⁵. Many projects get stuck in proof-of-concept mode or are abandoned before they can scale. In fact, about 72% of IoT initiatives never progress beyond the pilot (PoC) phase into full production².
Low Full-Success Rates: Conversely, only a 20 to 30% minority of IoT projects are deemed fully successful. Cisco’s oft-cited survey found just 26% of IoT projects were completed successfully (with the rest stalled or failing)³. Microsoft’s IoT Signals report similarly observed roughly 30% of IoT projects fail at the PoC stage, usually due to high implementation costs or unclear business benefits¹. Even when projects do launch, many underperform – an IDC study of industrial firms found 31% of IoT/IIoT projects yielded only “minimal payback,” failing to meet their expected ROI⁶.
Signs of Improvement: On a positive note, there are indications that IoT success rates are slowly improving as organizations gain experience. One 2023 research study (Beecham Research) reports a "28% improvement" in IoT project success metrics compared to 2020⁷. This suggests companies are learning from past failures and adopting better practices.

Nonetheless, IoT projects remain risky – success is far from guaranteed, and partial or outright failures are still common across industries.

📉 75% of IoT projects fail to meet expectations.
🔐 40% of companies cite security as the #1 challenge.
💸 75% of projects take twice as long and run 45% over budget.

Top Reasons for IoT Project Failures

Why do so many IoT initiatives struggle or fail? Recent surveys and expert analyses point to several recurring problem areas that derail IoT projects:

Unclear Objectives: A leading cause of failure is starting an IoT project without well-defined business goals or success criteria. IoT projects are inherently complex, and without clear objectives, they are destined to fail². Many organizations dive into IoT for the technology’s sake rather than to solve a specific business problem. This leads to stakeholder misalignment and unclear ROI. Only the few companies that set specific, measurable goals and KPIs for IoT from the outset tend to “enjoy fruitful results”². In contrast, vague goals or an undefined value proposition will almost guarantee project failure.
Integration: IoT deployments often falter due to integration challenges – connecting diverse devices, platforms, and data streams is difficult. Enterprises struggle to merge IoT data with existing IT systems and analytics tools. Studies show that less than half of IoT data (structured) gets actively used in decision-making, and under 1% of unstructured IoT data is ever analyzed². This means many projects collect data but cannot integrate or utilize it effectively, yielding limited insights. Poor data integration and siloed systems thus lead to disappointing outcomes. Ensuring seamless device, data, and software integration is crucial for IoT success, and failure to do so is a top reason projects under-deliver.
Scalability: Moving an IoT proof-of-concept to a reliable, scaled deployment is a major hurdle. Many projects work in the lab but struggle with real-world scalability. A Gartner study found about 30% of IoT projects fail due to scalability problems – solutions that worked with dozens of devices cannot handle thousands². IoT systems must be designed to handle growth in device count, data volume, and users, but companies often underestimate this. Complexities multiply when scaling (network strain, data overload, device management issues), resulting in stalled rollouts. Without a scalable architecture and plan for growth, IoT pilots can collapse when facing real-world scale.
Security and Data Governance: Security vulnerabilities are a frequent IoT project killer. IoT expands the cyber attack surface, and many projects fail to implement adequate security controls for devices and data. In a recent industry survey, 40% of organizations cited security concerns as the #1 challenge holding back IoT initiatives⁸. Weak security can lead to breaches or compliance issues that halt a project. Security is a major hurdle in successfully launching and managing IoT projects, and must be addressed end-to-end (device, network, cloud)². Projects that treat security as an afterthought often face disaster – for instance, unresolved data privacy risks can prevent an IoT solution from ever going live. Robust security and privacy measures are therefore prerequisite for IoT success, and their absence is a top reason for failure.
Timeline Delays and Budget Overruns: IoT implementations frequently take far longer and cost more than planned, leading to stakeholder fatigue or funding cuts. One study estimates 75% of IoT projects take twice as long as initially scheduled². Extended development and testing cycles can cause loss of momentum and executive support. Moreover, large IoT projects often run ~45% over budget while delivering 56% less value than anticipated². Such cost overruns and lower-than-expected ROI make it hard to justify continuing. These timeline and budget issues are often linked to the factors above (underestimated complexity, integration woes, etc.). If the project’s costs spiral or value remains unproven, it may be deemed a failure. Careful planning, agile execution, and controlling scope are needed to avoid this fate.
Skill Gaps and Organizational Challenges: IoT initiatives demand a mix of IT, engineering, data science, and business domain expertise – a combination many organizations lack. Talent shortages in IoT are widely reported; for example, half of IoT adopters say they don’t have enough skilled workers or training in IoT¹. Lacking in-house IoT skills often leads to design mistakes, integration errors, or reliance on misaligned third parties. Additionally, IoT projects cut across departments (IT, operations, R&D), so poor internal coordination or leadership support can doom projects. A survey by IoT World Today found lack of skilled personnel was the most-cited IoT challenge, and issues like insufficient management buy-in also impede progress[^8]. In short, organizations that don’t address the human and organizational factors – skills, alignment, change management – often see their IoT projects falter even if the technology pieces are in place.

Ensuring Success

Despite the challenges, a growing body of experience and research highlights how organizations can improve IoT project success rates. Several key success factors and emerging trends are making a difference in 2025.

Clarifying Objectives

One of the most cited reasons for failure is unclear goals. But what does success look like? Successful IoT programs start with a clear strategy and often use an agile, phased rollout. Rather than attempting a “big bang” deployment, leading adopters pilot their IoT solutions in stages, learn, and iterate.

IoT business decision makers may not be fully aware of the technological capabilities and limitations of IoT and their chosen IoT platform — so committing up front to absolute requirements, scope and architecture can be difficult. A vital tool to help answer the unknowns is the ability to quickly prototype the entire solution, expose key design concepts to reduce risk and then evolve the concept.

An incremental approach greatly increases the odds of success – a recent survey found companies using phased IoT rollouts experience ~40% higher success rates than those going for full-scale launches at once⁹.

To address this, the EmbedThis Ioto device management meta-platform provides a rapid prototyping environment that helps business stakeholders quickly visualize and refine their IoT strategy — long before committing to large-scale implementation. With Ioto, developers can instantiate device clouds in minutes, emulate target hardware, and create custom apps rapidly — all before committing to a particular architectural implementation. This allows key design assumptions and constraints to be tested early and at low cost.

Easing Integration

Creating and integrating all the necessary components for an IoT solution is challenging. It typically involves designing, creating and integrating device hardware, firmware, device clouds, networking, cloud services, data modeling, analytics and device apps.

A trend among successful IoT deployments is the use of modular, scalable products and services to simplify development. Rather than reinventing the wheel, companies are leveraging proven IoT platforms such as AWS IoT and EmbedThis Ioto. These handle the heavy lifting of connectivity, security, data ingestion, storage, integration, analytics and visualization. This can drastically cut time-to-market and avoid integration pitfalls. Gartner research indicates organizations using standardized IoT communication protocols lower their integration costs by up to 30% and improve deployment time by 25%⁹.

EmbedThis Ioto is a meta-platform that builds upon the AWS IoT platform to offer an end-to-end device management solution. Ioto provides device agent, device cloud, device builder service and device apps. All components are designed to work together and to fully integrate with the underlying AWS IoT platform. This reduces — and often eliminates — the need for lengthy integration cycles and greatly reduces development cost and integration risk.

Ensuring Scalability

Scalability issues often surface late in an IoT project when design and architectural decisions are well "baked in". Fixing these issues by changing design is costly in both time and budget.

Ioto has been designed from the beginning to scale to the very largest installations. Device populations up to and beyond 10 million devices are fully supported. By using the huge scale of the underlying AWS IoT platform and by utilizing built-for-scale technologies such as AWS DynamoDB and serverless Lambda compute, the Ioto platform can scale to support the largest IoT workloads.

Designing for Security

Given that security is a make-or-break factor, successful IoT initiatives now take a “secure by design” approach. This means building in encryption, authentication, network security, and continuous monitoring from the start, rather than bolting security on later. Companies that achieve IoT success tend to proactively address security and privacy – e.g. using end-to-end encryption, robust identity management for devices, and thorough testing for vulnerabilities². This reduces the risk of costly breaches or compliance failures down the road. Similarly, designing for reliable connectivity and device management is crucial.

Unfortunately, the typical IoT attack surface is quite large, spanning from device hardware, software, networking, and cloud services to end-user device apps. Consequently, the effort and skill-set required to fully and comprehensively design a secure IoT solution is a significant obstacle.

Ioto addresses this gap by providing a tested, end-to-end device management solution and applies proven security design patterns across the stack in all components including: the device agent, device communications, cloud service, device data storage and end-user apps. With Ioto, your data is stored in a dedicated private AWS database in a local AWS IoT region of your choosing. Your device data never transits another service or network and goes directly from the device to device cloud. Device data is reliably stored with 6 replicas and full point-in-time backups. Your compute logic runs in 3 AWS availability zones for maximum uptime.

Bridging the Skills Gap

The human factor plays a critical role in IoT success. Leading initiatives prioritize assembling the right team and forging strong partnerships. Since the expertise required to deliver a robust IoT solution is often scarce or unavailable in-house, relying on existing, well-tested technologies can significantly reduce development time and minimize the impact of skills gaps.

By adopting the Ioto device management platform, you can focus your team’s efforts on the areas most aligned with your core business, reducing the need to build deep expertise across the entire IoT stack.

Conclusion and Outlook

In 2025, IoT projects are gradually improving as industries learn from early missteps, but the failure rate remains unacceptably high. Only about a quarter of IoT projects today can be counted as unqualified successes, with the rest either falling short of goals or never fully launching³. The top pitfalls – unclear purpose, integration woes, scalability, security, cost overruns, and skill gaps – have become well recognized.

IoT project outcomes can be significantly improved by applying the lessons of recent failures.
Organizations that adopt end-to-end solutions like Ioto dramatically reduce complexity, risk, and time-to-market. The path to a successful IoT deployment starts with choosing the right foundation.

References

Ioto Device Management for Volume Device Builders

Michael O'Brien — Fri, 10 May 2024 06:32:07 +0000

The proliferation of Internet of Things (IoT) devices in our daily lives has led device manufacturers to seek ways to incorporate cloud-based management into their products. However, building a cloud-based device management solution is a complex task that requires expertise in various domains, including embedded development, communications, cloud computing, and user interface (UI) and user experience (UX) design.

To create a solution using Amazon Web Services (AWS) IoT, for instance, requires knowledge of numerous AWS services. Therefore, it can be challenging to develop a cloud-based device management solution that is secure, scalable, and cost-effective to maintain.

So we created Ioto to be the most secure, scalable IoT solution for volume device builders.

Creating Ioto

With over two decades of experience in device management and the development of device agents, EmbedThis has a wealth of knowledge in creating secure and effective solutions for device builders of all sizes.

Our GoAhead and Appweb embedded web servers are two of the most popular device agents and have been widely used in the industry. As cloud-based remote management becomes increasingly important, we wanted to create the best possible platform for scalable and secure device management both locally and via the cloud.

The result is Ioto, a comprehensive cloud-based solution for device-based and cloud-based management that includes an embedded device agent, cloud device management service, device builder portal, and user device manager port.

What is Unique about Ioto?

The EmbedThis Ioto solution is a complete, end-to-end device management solution that includes an embedded device agent, cloud device services, builder portal, and user device managers. It is built upon the reliable and secure AWS IoT infrastructure, ensuring that it is scalable, cost-effective, and secure for device management. Ioto offers a comprehensive platform for managing devices locally and remotely via the cloud.

The Ioto Solution

Ioto includes the following core components:

Ioto Device Agent
Ioto Device Manager
Device Builder Portal
Device Clouds
Ioto Device Service

The Ioto Device Agent is embedded in devices and communicates securely with the Ioto Device Manager for local management or via the Ioto Cloud Service for cloud-based management.

The Ioto Device Manager is a customizable, white-labeled device management platform for users to monitor and manage their devices. It can be branded with the your own logo and design elements to create a seamless experience for your customers. It can be embedded in your device for local device-based management or hosted in the cloud for cloud-based management.

The Device Builder Portal is a tool that allows device manufacturers to design, configure, and manage their device management solutions. It is used to subscribe and download device agents, create Device Clouds and manage manufactured devices.

Ioto Device Agent

Embedthis Ioto is a small but powerful embedded agent for local and remote device management. It boasts impressive speed and a comprehensive range of management protocols and capabilities.

Ioto includes an HTTP web server, embedded database, MQTT client, HTTP client, JSON parsing, AWS IoT cloud integration, easy provisioning, and OTA upgrading. It can be used for local management through its embedded web server or integrated with the cloud through comprehensive AWS IoT integration. Ioto offers a versatile and flexible solution for managing devices in a variety of environments.

In addition to its range of features and capabilities, Ioto also has a very small memory footprint of only 200K of code, making it ideal for use on Linux and FreeRTOS systems. It can also be easily ported to other platforms, providing flexibility and versatility for device management on a variety of systems.

Agent Components

Ioto provides the following components:

HTTP/1.1 server with dynamic rendering, authentication, cookies, sessions and file upload
HTTP/1.1 client
MQTT/3.1.1 client
Embedded database
JSON/5 parser and query engine
Transport Layer Security (TLS/SSL) with ALPN support
AWS IoT Integration with AWS IoT and AWS DynamoDB
AWS service integration with S3, Lambda, Kinesis and CloudWatch
Transparent database synchronization to AWS DynamoDB (like Global Tables)
Safe, secure runtime core
Streamlined certificate and key provisioning
Over-The-Air updates and upgrades
User authentication management
Complete documentation and samples
Full Source code

Ioto Device Manager

The Ioto solution provides a configurable web app from which your users can monitor and manage their devices.

The Device Manager can be embedded in your device for local management or deployed from the cloud for cloud-based management. The Builder portal is used to create and configure device managers.

The Ioto Manager is a generic (white-label) cloud-based device manager that can be extensively customized to manage your devices with your logo, product name, color and font theme, device data and device specific screens and interface including browser-based and cloud-side custom logic.

The Ioto Manager is extremely flexible, however, there are limits, and you may eventually want to create your own manager application from the ground up, that uses the Ioto APIs to provide a bespoke management experience for your devices.

Why AWS IoT for Cloud-based Management?

EmbedThis has chosen AWS IoT as the foundation for its Ioto cloud management solution for several key reasons:

Scalability: AWS IoT can handle a large number of devices and handle the data generated by them, making it suitable for use in large-scale IoT deployments.
Security: AWS IoT has built-in security measures such as encryption, authentication, and access controls to protect device data and communication. Further, AWS supports over two dozen regions, so your data can be hosted in your AWS account near you.
Integration with other AWS services: AWS IoT can be easily integrated with other AWS services such as Amazon Kinesis, Amazon S3, and Amazon Machine Learning, allowing for further processing and analysis of device data.
Cost-effectiveness: AWS IoT offers a pay-as-you-go pricing model, allowing users to only pay for the resources they consume, making it cost-effective for device management.
Reliability: AWS has a proven track record of reliability, with multiple availability zones and disaster recovery measures in place to ensure smooth operation of its services.

With Ioto, you get the best of both worlds. A complete end-to-end IoT solution and the rock-solid foundation of AWS IoT.

Want More?

To learn more about EmbedThis Ioto, please read:

Parallelism via Fiber Coroutines

Michael O'Brien — Tue, 19 Sep 2023 05:08:39 +0000

Server applications such as web servers often need to service multiple requests in parallel. This can be achieve by using threads, event callbacks or via fiber coroutines. While fiber coroutines are less well known and understood, they have some compelling reasons to be uses instead of threads or callbacks.

A fiber coroutine is a code segment that runs with its own stack and cooperatively yields to other fibers when it needs to wait. Fibers can be viewed as threads, but only one fiber runs at a time. For Go programmers, fibers are similar to Go routines, while for JavaScript developers, fibers are comparable to async/await.

The Ioto embedded web server's core uses fiber coroutines to serve multiple requests in parallel. Ioto is based on a single-threaded fiber coroutine architecture that employs a non-blocking, event-driven design capable of handling numerous inbound and outbound requests simultaneously with minimal CPU and memory resources. Ioto simplifies programming by eliminating the complexity of threads and the inelegance of event callbacks through the use of fiber coroutines.

Ioto's fibers are integrated into the I/O system, enabling parallelism to be effortlessly supported. All Ioto services support fibers, making your user extension code straightforward, easy to debug, and maintainable in the long term. You can use a straight-line procedural programming model to read and write sockets, issue HTTP client requests, send MQTT messages, or respond to web server requests.

Models of Parallelism

To implement parallelism in an application, a developer has three choices:

Threads
Non-blocking APIs with callbacks
Fiber coroutines

Threads

Programming with threads can be appealing at first, however a multithreaded design can be problematic. Subtle programming errors due to timing related issues, multithread lock deadlocks and race conditions can be extraordinarily difficult to detect and diagnose. All too often, they appear only in production deployments.

Although some developers excel in creating multithreaded designs, others may struggle when tasked with maintaining complex threaded code and debugging subtle race conditions and issues. Over time, a design that initially seemed reasonable can become increasingly challenging to maintain and support.

Callbacks

An alternative method for implementing parallelism involves the use of non-blocking APIs coupled with callbacks, which are often easier to test and debug compared to threaded designs. However, this approach often leads to decreased code quality due to the prevalent "callback-hell" phenomenon. In such cases, relatively simple algorithms become obfuscated as they are dispersed across cascading callbacks. This problem is especially pronounced in C or C++ coding designs that lack inline lambda functions for simplification. Consequently, linear algorithms are fragmented across multiple functions, and clear algorithms become increasingly difficult to decipher.

Fiber Coroutines

An appealing alternative for implementing parallelism is through the use of Fiber Coroutines. A Fiber coroutine refers to code that runs with its stack and cooperatively yields to other fibers when it needs to wait.

Fibers can be thought of as threads, but only one fiber runs at a time, eliminating the need for thread locking or synchronization. For Go programmers, fibers are akin to Go routines, while for JavaScript developers, fibers are similar to async/await.

By allowing programs to overlap waiting for I/O or other events with useful compute tasks, fibers achieve parallelism without the complexities involved in other methods.

Fibers address the primary issue with multi-threaded programming where multiple threads access the same data at the same time, requiring complex locking to safeguard data integrity. Furthermore, they resolve the primary problem with non-blocking callbacks by enabling a procedural straight-line coding style.

Although not flawless, fibers provide an efficient solution for achieving parallelism. It may not allow full utilization of all the CPU cores of a system within one program. However, for embedded device management, this is generally not a significant concern. Since device management applications are usually secondary to the primary role of the device, they should not monopolize the CPU cores of the device.

Parallelism Compared

Consider a threaded example:

int count = 0;
pthread_mutex_t mutex;

void increment() {
    pthread_mutex_lock(&mutex);
    count = count + 1;
    pthread_mutex_unlock(&mutex);
}

void getCount() {
    int c;
    pthread_mutex_lock(&mutex);
    c = count;
    pthread_mutex_unlock(&mutex);
    return c;
}

Now consider the fiber solution:

int count = 0;

void increment() {
    count = count + 1;
}

void getCount() {
    return ccount;
}

Since only one segment of code is executing at any one time, there is no possibility of fiber collisions.

Callback Example

When implementing parallelism with callbacks, applications must employ non-blocking I/O. While blocking I/O is simpler, it prohibits the application from performing any other function while waiting for I/O to complete.

For instance, consider an application that must execute a REST HTTP request to retrieve some remote data. While waiting for the request to complete, the application is blocked and cannot perform any other task for several seconds. Non-blocking I/O resolves this issue, but creates another problem known as "callback hell".

Consider this pseudo-example:

//  Issue a request and invoke the onData callback on completion
httpFetch("https://www.example.com", onData)
return;

//  First Callback
static void onData(HttpResult *result)
{
    if (!result) {
        //  Invoke another request
        httpFetch("https://www.backup.com/);
    }
}

//  Second Callback
static void onComplete(HttpResult *result)
{
    //  Now we done and can process the result
}

As the level of callback nesting increases, the code's intended purpose rapidly gets obscured.

The alternative Ioto code using fiber coroutines would look like this:

int data = urlGet("https://www.example.com");
if (!data) {
    data = urlGet("https://www.backup.com/");
}

The calls to urlGet will yield and other fibers will run while waiting for I/O. When the request completes, this fiber is transparently resumed and execution continues.

Code based on Fibers is more straightforward to code, debug, and maintain.

When transitioning Ioto from callbacks to fibers, several of our algorithms reduced code lines by over 30%.

Ioto Fibers in Practice

In practice, when working with Ioto, there is usually no need to explicitly program fiber yielding or resuming. The Ioto socket APIs are fiber-aware and will handle the yielding for you.

All Ioto services, including the web server, Url client, MQTT client, and AWS services, feature async APIs that are fiber-aware and will yield and resume automatically.

For example:

char buf[1024];

while ((nbytes = rSocketRead(sock, buf, sizeof(buf))) > 0) {
    printf("Got body data %.*s\n", (int) nbytes, buf);
}

Ioto I/O API

Ioto builds fiber support into the lowest layer of the "R" portable runtime. The following APIs support automatic fiber yielding:

The rReadSocket and rWriteSocket APIs will yield and resume the current fiber as required, allowing other fibers to continue running. It is important to note that only one fiber will execute at a time.

Fiber API

Ioto also supports a low level fiber API so you can construct your own fiber-enabled primitives.

Use rYieldFiber to yield the CPU and switch to another fiber. You must make alternate arrangements to call rResumeFiber when required.

Use rSpawnFiber to create a new fiber and immediately switch to it. For example:

void myFiberFunction(void *arg) {
    //  code here runs inside a fiber
}
rSpawnFiber(myFiberFunction, arg);

Integrating with External Services

But what should you do if you need to invoke an external service that will block?

You have two alternatives:

Use Non-Blocking APIs
Use threads

Using Non-Blocking with External Services

Ioto provides a flexible centralized eventing and waiting mechanism that can support any service that provides a select() compatible file descriptor.

If the external service has a non-blocking API and provides a file descriptor that is compatible with select or epoll, you can use the Ioto runtime wait APIs to be signaled when the external service is complete.

To wait for I/O on a file descriptor, call rAllocWait to create a wait object and rSetWaitHandler to nominate an event function to invoke.

For example:

wait = rAllocWait(fd);
rSetWaitHandler(wait, fn, arg, R_READABLE);

The nominated function will be run on a fiber coroutine when I/O on the file descriptor (fd) is ready.

Using Threads with External Services

The other option is to create a thread. However you must take care to properly yield the fiber first. The runtime provides a convenient rSpawnThread API that will do this for you. It will create a thread, yield the current fiber and then invoke your threadMain. When your threadMain exits, it will automatically resume the fiber.

For example:

rSpawnThread(threadMain, arg);

static void threadMain(void *arg)
{
    data = getFromExternalService();
    return data;
}

Manual Yield and Resume

Though unlikely, you may have a need to manually create fibers and yield and resume explicitly.

The APIs for this are: rAllocFiber, rYieldFiber and rResumeFiber.

See the Runtime API for more details.

Summary

Ioto eliminates the complexity of threads and verbosity of callbacks by using fiber coroutines. The result is a simple, highly efficient design that simplifies implementing and debugging IoT and embedded services.

Want More Now?

To learn more about EmbedThis Ioto, please read:

CustomMetrics -- Simple, Cost-Effective Metrics for AWS

Michael O'Brien — Tue, 19 Sep 2023 04:49:14 +0000

AWS CloudWatch offers metrics for monitoring specific aspects of your applications. However, AWS custom metrics can become costly when updated or queried frequently, with each custom metric costing up to $3.60 per metric per year, along with additional expenses for querying. If you have a significant number of metrics or high dimensionality in your metrics, this could result in a substantial CloudWatch Metrics bill.

On the other hand, CustomMetrics presents a cost-effective alternative metrics API that is considerably more budget-friendly and efficient compared to standard CloudWatch metrics.

The cost savings achieved by CustomMetrics are primarily due to its focus on providing only the most recent period metrics, such as those from the last day, last month, last hour, last 5 minutes, and so on. These metric timespans are also fully configurable. This approach ensures that each metric can be saved, stored, and queried with minimal cost.

AWS CloudWatch Metrics

Frequently, users complain that CloudWatch is one of the most expensive parts of their AWS bill. For those users that employ custom metrics with high update frequency or high dimensionality, this can quickly translate into a large bill.

CloudWatch charges based on the number of metrics sent to the service and the frequency of updating or querying metrics. Your bill will increase as you send more metrics to CloudWatch and make API calls more frequently. For a regularly updated or queried metric, you will pay $0.30 per metric per month for the first $3,000 per month.

AWS metrics are expensive because they store metrics with arbitrary data spans. You can query metrics for any desired period (with decreasing granularity as the metrics age). On the positive side of the ledger, you can use CloudWatch EMF log format to emit metrics from Lambda's without invoking an API. But you still pay for maintenance of the metric.

CustomMetrics Alternative

CustomMetrics foregoes the option for arbitrary date queries and provides "latest" period metrics only. When using CustomMetrics, you request data for specific recent time spans such as the last "5 minutes," "hour," "day," "week," "month," or "year." You have the flexibility to configure these "last" time spans according to your preferences, but they are always based on the current time. For example, you could record metrics for the last "minute" and "15 minutes".

In exchange for its exclusive emphasis on the most recent metrics, CustomMetrics can store and retrieve metrics at a significantly lower cost compared to CloudWatch custom metrics.

CustomMetrics Features

CustomMetrics is a NodeJS library designed to emit and query custom metrics for AWS applications. Is offers the following features:

Simple one line API to emit metrics from any NodeJS TypeScript or JavaScript app.
Similar metric model to AWS CloudWatch for supporting namespaces, metrics, dimensions, statistics and intervals.
Computes statistics for: average, min, max, count and sum.
Computes P value statistics with configurable P value resolution.
Supports a default metric intervals of: last 5 mins, hour, day, week, month and year.
Configurable custom intervals for different metric timespans and intervals.
Fast and flexible metric query API.
Query API can return data points or aggregate metric data to a single statistic.
Scalable to support many simultaneous clients emitting metrics.
Stores data in any existing DynamoDB table and coexists with existing app data.
Supports multiple services, apps, namespaces and metrics in a single DynamoDB table.
Extremely fast initialization time.
Written in TypeScript with full TypeScript support.
Clean, readable, small, TypeScript code base (~1.3K lines).
SenseDeep support for visualizing and graphing metrics.
DynamoDB Onetable support CustomMetrics for detailed single table metrics.

Database

CustomMetrics stores each metric in a single, compressed DynamoDB item. Each metric stores the optimized data points for the metric's timespans. The default spans are 5 mins, 1 hour, 1 day, 1 week, 1 month and 1 year. But these can be configured for each CustomMetric instance.

Emitting a metric via the emit API will write the metric via a DynamoDB item update. Multiple simulataneous clients can update the same metrics, and CustomMetrics will ensure no data is lost.

If optimized metric buffering is enabled, metric updates may be aggregated according to your buffering policy to minimize the database write load.

Quick Tour

Here is a quick tour of CustomMetrics demonstrating how to install, configure and use it in your apps.

First install the library using npm:

npm i custom-metrics

Import the CustomMetrics library. If you are not using ES modules or TypeScript, use require to import the library.

import {CustomMetrics} from 'CustomMetrics'

Next create and configure the CustomMetrics instance by nominating the DynamoDB table and key structure to hold your metrics.

const metrics = new CustomMetrics({
    table: 'MyTable',
    region: 'us-east-1',
    primaryKey: 'pk',
    sortKey: 'sk',
})

Metrics are stored in the DynamoDB database referenced by the table name in the desired region. This table can be your existing application DynamoDB table and metrics can safely coexist with your app data.

The primaryKey and sortKey are the primary and sort keys for the main table index. These default to 'pk' and 'sk' respectively. CustomMetrics does not support tables without a sort key.

If you have an existing AWS SDK V3 DynamoDB client instance, you can use that with the CustomMetrics constructor. This will have slightly faster initialization time than simply providing the table name.

import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
const dynamoDbClient = new DynamoDBClient()

const metrics = new CustomMetrics({
    client: myDynamoDbClient,
    table: 'MyTable',
    region: 'us-east-1',
    primaryKey: 'pk',
    sortKey: 'sk',
})

Emitting Metric Data

You can emit metrics via the emit API:

await metrics.emit('Acme/Metrics', 'launches', 10)

This will emit the launches metric in the Acme/Metrics namespace with the value of 10.

A metric can have dimensions that are unique metric values for specific instances. For example, we may want to count the number of launches for a specific rocket.

await metrics.emit('Acme/Metrics', 'launches', 10, [
    {rocket: 'saturnV'}
])

The metric will be emitted for each dimension provided. A dimension may have one or more properties. A metric can also be emitted for multiple dimensions.

If you want to emit a metric over all dimensions, you can add {}. For example:

await metrics.emit('Acme/Metrics', 'launches', 10, [
    {}, 
    {rocket: 'saturnV'}
])
await metrics.emit('Acme/Metrics', 'launches', 10, [
    {}, 
    {rocket: 'falcon9'}
])

This will emit a metric that is a total of all launches for all rocket types.

Query Metrics

To query a metric, use the query method:

let results = await metrics.query('Acme/Metrics', 'speed', {
    rocket: 'saturnV'
}, 86400, 'max')

This will retrieve the speed metric from the Acme/Metrics namespace for the {rocket == 'saturnV'} dimension. The data points returned will be the maximum speed measured over the day's launches (86400 seconds).

This will return data like this:

{
    "namespace": "Acme/Metrics",
    "metric": "launches",
    "dimensions": {"rocket": "saturnV"},
    "spans": [{
        "end": 946648800,
        "period": 300,
        "samples": 10,
        "points": [
            { "sum": 24000, "count": 19, "min": 1000, "max": 5000 },
        ]
    }]
}

If you want to query the results as a single value over the entire period (instead of as a set of data points), set the accumulate options to true.

let results = await metrics.query('Acme/Metrics', 'speed', {
    rocket: 'saturnV'
}, 86400, 'max', {accumulate: true})

This will return a single maximum speed over the last day.

To obtain a list of metrics, use the getMetricList method:

let list: MetricList = await metrics.getMetricList()

This will return an array of available namespaces in list.namespaces.

To get a list of the metrics available for a given namespace, pass the namespace as the first argument.

let list: MetricList = await metrics.getMetricList('Acme/Metrics')

This will return a list of metrics in list.metrics. Note: this will return the namespaces and metrics for any namespace that begins with the given namespace. Consequently, all namespaces should be unique and not be substrings of another namespace.

To get a list of the dimensions available for a metric, pass in a namespace and metric.

let list: MetricList = await metrics.getMetricList('Acme/Metrics', 'speed')

This will also return a list of dimensions in list.dimensions.

Metrics Tenants

You can scope metrics by chosing unique namespaces for different applications or services, or by using various dimensions for applications/services. This is the preferred design pattern.

You can also scope metrics by selecting a unique owner property via the CustomMetrics constructor. This property is used, in the primary key of metric items. This owner defaults to 'default'.

const cartMetrics = new CustomMetrics({
    owner: 'cart',
    table: 'MyTable',
    primaryKey: 'pk',
    sortKey: 'sk',
})

References

Participate

All feedback, discussion, contributions and bug reports are very welcome.

Use Case

We've used Custom Metrics extensively in our EmbedThis Ioto IoT middleware service.

Contact

You can contact me (Michael O'Brien) on Twitter at: @mobstream, and read my Blog.

How to debug serverless apps

Michael O'Brien — Tue, 29 Mar 2022 03:56:52 +0000

Debugging serverless apps is different and difficult.

If you have not properly prepared, debugging some requests may be impossible with your current level of instrumentation and tooling.

So what is the best way to debug serverless apps and serverless requests?

The blog post describes how we debug the serverless backend for our SenseDeep Serverless Developer Studio service and what tools we use with our NodeJS serverless apps.

Some of the suggestions here may not be suitable for all sites, but I hope they give you ideas to improve your ability to debug your serverless apps.

But first, some background.

Serverless Computing Changes Things

The way enterprises design, debug, and ship applications changed forever when serverless computing arrived on the scene. The introduction of serverless allowed developers to build and ship much faster, which in turn allows them to concentrate on coding rather than maintenance, auto-scaling, and server provisioning.

Serverless is now mainstream and it offers many compelling advantages, but debugging serverless apps is still a tough problem as the required tools have not kept pace.

What is different about serverless debugging?

Debugging a server-based app (monolith) is well understood and assisted by a suite of refined tools that have been created over decades of evolution. Debugging can be performed using IDEs either locally or remotely over SSH and other protocols. Importantly, server state is typically persisted after requests to permit live or after-the-fact inspection and debugging.

Serverless is not so lucky

Serverless is different. It is often more complex with multiple interacting, loosely coupled components. Instead of a monolith, it is more like a spider's web and consequently harder to follow a single request with no ability for live debugging or setting breakpoints to intercept code execution.

Debugging serverless via remote shell or execution is not possible. Serverless requests are ephemeral and request state is not available after execution. Once a request has been completed, there is no state left behind to examine.

So when a serverless request errors, it often fails silently or any record is buried in a mountain of log data in CloudWatch. You will not be proactively notified and if you are informed by the end-user, finding the failed request is a "needle in a haystack" type of problem.

Furthermore, comprehensive local emulation of cloud environments is either not possible or has severe limitations. As cloud services become more evolved, local emulation is becoming increasingly limited in scope and application.

So we need a different technique to debug serverless compared to server-based app debugging.

Serverless Debugging Technique

The primary means of debugging serverless apps is via detailed, intelligent request and state logging, paired with log management to correlate log data and quickly query to locate events of interest.

Whether your code is under development, is being tested as part of unit or integration tests, or in production, detailed logging is the foundation for you to see and understand what is actually happening in your code for each request.

Detailed, intelligent logging is the foundation for serverless observability.

Observability

Observability is the ability to understand your system's unknown unknowns. i.e. the ability to understand not just expected errors, but also unforeseeable conditions. It is not enough to just log simple, plain text error messages.

Our understanding of all the possible the failure modes of serverless apps is limited by the extent of our knowledge today. Serverless apps can fail in many surprising ways that we may not yet understand and cannot foresee.

So to meet the debugging needs of tomorrow, you need to log much more information and context than you might realize. Your app must emit sufficient state and information to permit debugging a wide range of unexpected conditions in the future without having to stop and redeploy code.

Fortunately, this is easy to do.

How to Log? Extensively.

Often developers add diagnostic logging as an after thought via sprinkled console.log messages.

console.log("Oops, bad thing happened");

However, using console.log to emit simple messages is almost useless in achieving true Observability. Instead, what is required is the ability to emit log messages in a structured way with extra context that captures detailed request state.

Every time a developer uses console.log an angel cries in heaven.

In addition to the log message, log events should emit additional context information. To achieve this, the output format must be structured. JSON is an ideal format as most log viewers can effectively parse JSON and understand the context emitted.

For example:

log.error(`Cannot authenticate user`, {user, auth, req});

This would emit something like:

{
    "timestamp": "2022-03-26T06:41:59.216ZZ",
    "message": "Cannot authenticate user",
    "type": "error",
    "user": { ... },
    "auth": { ... },
    "req": { ... },
    "stack": [ /* stack trace */ ]
}

There are many log libraries that are capable of emitting structured log context data. For SenseDeep, we use the SenseLogs library which is an exceptionally fast logging library designed for serverless. It has a flexible, simple syntax that makes adding detailed log events easy.

SenseLogs emits log messages with context and stack traces in JSON. It can also be dynamically controlled as to which log messages are emitted at runtime without having to redeploy code.

Here is a mock Lambda which initializes SenseLogs and emits various sample log messages:

const log = new SenseLogs()

exports.handler = async (event, context) => {
    log.addTraceIds(event, context)

    let body = JSON.parse(event.body)
    log.info('Request start', {body, event, context})

    log.info('New user login', {user, auth})
    log.debug('Queue Stats', {q, backlog})
    log.fatal('Unexpected exception with cart', {err, cart})

    log.emit('Beta', 'Performance metrics for beta release features', {
        metrics
    })
}

SenseLogs emits messages into channels such as info, debug, error. These channels can be enabled or disabled via Lambda environment variables. In this way, SenseLogs can dynamically scale up or down the volume of log data according.

Catch All Errors

It is very important to catch all errors and ensure they are reported. Lambdas should catch all exceptions and not rely on the default Lambda and language runtimes to catch errors. Employing a top-level catch can ensure that the exception report includes request state and correlation IDs.

Dynamic Logging

All log data has a cost and extensive logging can significantly impact the performance of your Lambdas and excessive log storage can cause rude billing surprises. Logging extensive state for all requests is typically prohibitively expensive.

A better approach is to enable a solid baseline of log data and to scale up and/or focus logging when required without redeploying code. This is called Dynamic Logging and it is important to be able to do this without the risk of redeploying modified code.

SenseLogs implements Dynamic Logging by listening to a Lambda LOG_FILTER environment variable. This variable may be set to a list of log channels to enable. By default, it is set to enable the following channels:

LOG_FILTER=fatal,error,info,metrics,warn

If the Lambda LOG_FILTER variable is modified, the next and subsequent Lambda invocations will use the adjusted channel filter settings. In this manner, you can scale up and down the volume and focus of your log messages without redeploying code.

You can use custom channels via log.emit to provide log coverage for specific paths in your code and then enable those channels on demand.

log.emit('feature-1', 'Unexpected condition, queue is empty', {
    /* state objects */ 
})

SenseLogs has two other environment variables which give more control over the log messages that are emitted.

LOG_OVERRIDE provides the list of channels to enable for a limited duration of time before automatically reverting to the LOG_FILTER channel set.
LOG_SAMPLE provides an additional list of channels to enable for a percentage of requests.

LOG_OVERRIDE is a convenient way to temporarily boost log messages that will automatically revert to the base level of logging when the duration completes. LOG_OVERRIDE is set to a Unix epoch timestamp (seconds since Jan 1 1970) when the override will expire, followed by a list of channels.

LOG_OVERRIDE=1626409530045:data,trace

LOG_SAMPLE is set to a percentage of requests to sample followed by a list of channels to add to the list of LOG_FILTER for sampled requests.

LOG_SAMPLE=1%:trace

See the SenseLogs Documentation for more details.

SenseDeep

You can modify environment variables with the AWS console:

But our SenseDeep serverless developer studio provides a much more convenient interface to manage Lambda LOG_FILTER settings for a single Lambda or for multiple Lambdas.

Base Logging

Your serverless apps should emit a good base of request state for all requests.

This should include the request parameters, request body and other high priority state information. The volume of log data should be low enough so that the cost of log ingestion and storage is not a significant burden.

Feature Logging

For critical code modules or less mature code, we include additional logging on custom channels that correspond to each module or feature. This can then be enabled via LOG_FILTER or LOG_OVERRIDE.

LOG_FILTER=fatal,error,info,metrics,warn,feature-1

or
LOG_OVERRIDE=1626409530045:feature-1

Sampled Logging

For a small percentage of requests, we log the full request state so that we always have some record of all the state values.
We use a custom SenseLogs channel called complete and we sample those requests at a low percentage rate.

log.emit('complete', { /* additional state objects */ })

LOG_SAMPLE=1%:complete

Correlation IDs

Serverless apps often have multiple Lambda functions or services that cooperate to respond to a single client request. Requests that originate with a single client request may traverse through many services and these services may be in different AWS regions or accounts.

A single request will often traverse through multiple AWS services such as: API Gateway, one more more Lambda functions, Kinesis streams, SQS queues, SNS messages, EventBridge events and other AWS services. The request may fan out to multiple Lambdas and results may be combined back into a single response.

To trace and debug an individual requests, we add a request ID to all our log events. This way we can filter and view a complete client request that flows though multiple Lambda services.

We add the trace IDs by using the SenseLogs API:

log.addTraceIds(event, context)

SenseLogs will extract the API Gateway request ID, Lambda request ID and X-Ray trace ID. SenseLogs will map these IDs to x-correlation-NAME SenseLogs context variables suitable for logging. The following variables are automatically detected and mapped:

x-correlation-api — API Gateway requestId
x-correlation-lambda — Lambda requestId
x-correlation-trace — X-Ray X-Amzn-Trace-Id header
x-correlation-extended — AWS extended request ID

SenseLogs will define a special variable 'x-correlation-id' that can be used as a stable request ID. It will be initialized to the value of the X-Correlation-ID header and if not defined, SenseLogs will use (in order) the API Gateway request or X-Ray trace ID.

SenseLogs also supports adding context state to the logger so you don't have to specify it on each logging API call. Thereafter, each log call will add the additional context to the logged event output.

Finally we define a client ID that is passed from the end user so we can track a "user request" through all AWS services that are invoked.

log.addContext({'x-correlation-client': body?.options.clientId})

Log Viewing and Querying

Once you have added detailed logging with correlation IDs to your Lambdas, you need an effective log viewer to display and query log events.

CloudWatch just won't cut it.

For our logging needs, we need a log viewer that can:

Aggregate not just log streams, but log groups into a unified set of events.
Aggregate log groups from different AWS accounts and regions.
Display log events in real-time with minimal latency.
Instantly query log events to isolate an individual request using correlation request IDs.
Manage dynamic log control to focus and scale up or down log data as required.

Our product, SenseDeep has such a log viewer that we use to monitor and debug the SenseDeep backend service.

Automated Alarms

Manually monitoring logs for app errors and signs of trouble is not scalable or reliable. An automated alarm mechanism when paired with detailed logging is the ideal platform to provide 24x7 oversight of your serverless apps.

We use such an automated approach. We configure multiple SenseDeep alarms to search for specific flags in log data that indicate errors or potential issues. If an alarm is triggered, an alert will immediately notify the relevant developer with the full context of the app event.

We instrument our code to not only capture and log errors, but also with "asserts" that test expected conditions and will emit log data that will trip alarms if unexpected conditions arise.

Triggering Alarms

To reliably trigger alarms, it is necessary to have a unique property value that the alarm mechanism can detect.

The SenseLogs logging library supports a flagging mechanism where log events to specific log channels can add a unique matchable "flag" to the log data. Alarms can then reliably test for the presence of such a flag in log data.

We use SenseLogs to add property flags for errors and asserts. By default, SenseLogs will add a FLAG_ERROR property for log events error() or fatal().

For other log channels, the flag option can be set to an map of channels, and the nominated channels will be flagged with the associated value string. For example:

new SenseLogs({flag: {warn: 'FLAG_WARN', error: 'FLAG_ERROR', custom: 'FLAG_CUSTOM'})
log.warn('Storm front coming')

This will emit:

{
    "message": "Storm front coming",
    "FLAG_WARN": true
}

We then create a SenseDeep alarm to match FLAG_ERROR and other FLAG_* patterns.

SenseDeep Alarms

We configure alarms for generic errors and exceptions as well as specific asserts and state conditions.

CloudWatch Alarms do not have the ability to efficiently monitor log data for specific log data patterns.

Fortunately, SenseDeep was built for this purpose.

SenseDeep developers create alarms to detect flagged errors when errors and unexpected conditions are encountered.

As SenseDeep ingests log data, it runs configured alarms to match app errors in real-time.

Alert Notifications

When an alarm triggers it generates an alert to notify the developer via email, SMS or other notification means.

The alert contains the full details of the flagged condition.

Clicking on the notification link will launch SenseDeep and display full details of the alert that triggered the alarm.

To see the log entries for the invocation, click on Goto Invocation.

If you want to see the preceding or subsequent log entries, click All Logs which will launch the log viewer homed to the invocation log data.

Serverless Goals

Using this serverless debugging pattern, we can detect problems early, typically before users are even aware of an issue.

The automated monitoring offloads the burden of "baby-sitting" a service and gives developers tools needed to ensure their apps are performing correctly.

CloudWatch Insights and X-Ray

You may wonder if you can use the native AWS services: CloudWatch and Xray to implement this serverless debugging pattern.

Short answer, you cannot.

CloudWatch does not aggregate log streams or groups. It is slow and only offers primitive query capabilities.

CloudWatch Insights does have the ability to search for correlation IDs in logs in a single region. But it is very, very slow and you cannot then see the request in context with the requests before or after the event. CloudWatch Insights is most useful for custom one-off queries, but not for repeated debugging efforts.

CloudWatch alarms cannot trigger alerts based on log data patterns. You can pair it with CloudWatch Insights, but this is a slow one-off solution and does not scale to process all log data for matching events.

Similarly, X-Ray has many good use cases, but because it only samples 1 request per second and 5% of additional requests, it cannot be used to consistently debug serverless applications. It is great for observing complete outages when an entire service or component is offline, but not for failing individual requests.

References

SenseDeep DynamoDB Data Browser

Michael O'Brien — Wed, 03 Nov 2021 00:30:18 +0000

The SenseDeep Data browser is a DynamoDB data browser and editor that can be used to query, manage and modify your DynamoDB data.

Developers are adopting DynamoDB single-table design patterns as the preferred design model where all application data is packed into a single-table. Combing disparate items with different attributes into one table can make browsing, navigating, organizing and viewing data obscure and difficult.

Managing single-table data and performance can often feel like you are peering at Assembly Language as it is hard to decode overridden keys and attributes manually. A new generation of tools is required.

SenseDeep can understand your single-table designs and make sense of your data and present items as intuitive application entities instead of raw data. The SenseDeep data browser is single-table "aware". This means SenseDeep can transform raw data items to present as application entities and fields.

Single-Table Data Browser

The SenseDeep data browser can query, manage and modify your table data. It supports browsing by scan, query or by single-table entities.

While the data browser can be used to browse and manage any DynamoDB table, it is turbo-charged when a schema describing your data is applied.

A scheme describes your application entities and their attributes. It specifies exactly how your data should be interpreted and what data is valid to store in the database table. You can import a schema from a JSON file (such as a OneTable schema) or you can define a schema using the SenseDeep single-table designer.

When armed with a schema, SenseDeep is able to "understand" your data and organize and present data items as application entities instead of raw, encoded DynamoDB items. The SenseDeep data browser intelligently displays and formats data according to your single-table schema. This not only makes your encoded single-table data easier to understand, but it guides and validates your changes and prevents schema-breaking and application breaking data changes.

The data browser supports DynamoDB scans, native queries or queries by application entity. Queries by application entity are normal DynamoDB queries but where SenseDeep understands which attributes are required to select and filter your data items.

Scanning

When scanning, select the index to scan and provide optional additional filtering attributes. Attributes can be combined using AND or OR operators. SenseDeep translates these instructions into DynamoDB scan commands. You can click on the Command button to see the generated DynamoDB command.

Querying

When querying, specify the index and partition key with optional sort key value. Additional filters may also be specified.

You can use sort key operations to select a single matching item with "Equal" or one or more items with the other Sort Key Operators from the pull down menu.

Query By Entity

When querying by Entity, you select the desired entity model and SenseDeep will select the appropriate attribute filters for that model.

SenseDeep understands the schema for the selected model and what are the required attributes to retrieve specific entity items. Sensedeep will guide you with the list of attributes for that entity model.

Saving Queries

Queries may be saved to your database table where they are persisted in the schema. The schema, saved in the database table, contains the entity definitions, saved queries and modeling data. This makes your table self-describing for 3rd party tools.

You can load and delete queries using the Queries button.

Data Visualization

SenseDeep groups, organizes and color-codes query results for maximum clarity.

Cells are color coded:

A cell is green if it can be edited. Click on the cell to modify the value (inline).
A cell is pink if the cells value is derived from a computed value template using other attributes as ingredients.
A cell has a wavy gray background if it is not relevant for this item (as defined by the schema).
A cell has a blue underline hot-link if it is a reference to another item and can be clicked to jump to the referenced item.

Columns are ordered with the index partition and sort key first, followed by the schema entity type attribute (if defined). After that, columns for all items are ordered alphabetically. If an item has many attributes, click the edit icon to display the item attributes vertically.

Hot Linked Items

Databases like DynamoDB have relationships between items. Just because it is a NoSQL database, does not mean there are no relationships. It just means there is little support in terms of joining tables, enforced foreign keys and data relationship integrity.

SenseDeep and OneTable bridge this gap by defining relationships between entities in the schema. SenseDeep will interpret these relationship links and highlight those in the browser with blue underlining.

You can quickly traverse related items by clicking on a blue hot link in the query results.

SenseDeep defines relationships between entity items using the schema "Reference" field property. This property specifies that a field refers to another entity item and which attributes are required to uniquely identify the target item.

Hot Link References

The format of the "Reference" field is:

Model:index:attribute-1=source-attribute-1,...

For example, the following Reference defines a link to an Account item:

Account:primary:id=accountId

This means: select the Account entity using the "primary" index and the "Account.id" attribute using the value of "accountId" from this item.

If you control-click (or Cmd-Click) on a hot-link SenseDeep uses the reference and determines the relevant query to locate the item and automatically fills the index, key and filter attributes accordingly. Queries are entered into your browser history so you can click the browser Back button to easily jump backwards to the original item.

You can also save these hot-linked queries using Queries -> Save.

Show Templates

An essential single-table design technique is decouple your keys from regular data attributes. This greatly enhances your ability to evolve your DynamoDB data going forward. SenseDeep and OneTable support this technique via value templates where key values are composed using templates that combine the values of other attributes at runtime.

It is often useful to view the value templates vs the calculated key values. The Templates toggle above the table switches between displaying the template values vs the actual data values.

Similarly, there may be some attributes that are designated as hidden. Changing the hidden toggle will display or hide these attributes.

Context Menu

In any data cell, you can right click to display the context menu for additional command options.

The options are:

Copy to copy the current item to the clipboard in JSON format.
Design Schema will jump to the SenseDeep single-table designer for the entity on which the current item is based.
Edit will open the editor slide-out panel for easy editing of the whole item.
Follow Reference behaves the same as Cmd-Click to follow a hot link reference.
Generate Mock Data option can be used when developing to generate sample data, such as email addresses or phone numbers.

Modifying Data

You can modify data inline by clicking on any green cell. Once modified, the Save button will be displayed above the items to persist the changes to the table.

To add a new item, click the Add Item button. You can select the desired entity model (if a schema is present) and it will intelligently prompt you for the appropriate attributes.

Modify Panel

You can also edit by clicking on the Edit pencil icon at the start of each item. This will display a slide out editor panel that will display only the attributes of the item, organized vertically.

The red cells have their value derived from other attributes and cannot be edited. You can modify the value template in the schema via the SenseDeep single-table designer.

If you change the partition key values either directly or indirectly be changing attributes that are used in a key value template, SenseDeep will atomically remove the old item and create the new item via a DynamoDB transaction.

When you click save, the changes are accepted, but you still must click the Save button on the query page to persist results to the table.

Import / Export

No man is an island and your data must be easy to export or import. SenseDeep provides an "Export to AWS Workbench" option. This will export your schema and data items into a WorkBench model. This model can also be imported by the Dynobase app.

When exporting, you should limit the amount of data you export. WorkBench models are for development and are designed for limited data sets. You can also export in a JSON backup format.

When importing, you can import a model to an existing (empty) table, or you can dynamically create a new table to hold the imported model.

Summary

Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.

Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.

SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.

More?

Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App or learn more at https://www.sensedeep.com.

About SenseDeep

SenseDeep is an observability platform for AWS developers to accelerate the delivery and maintenance of serverless applications.

SenseDeep helps developers through the entire lifecycle to create observable, reliable and maintainable apps via an integrated serverless developer studio that includes deep insights into how your apps are performing.

To try SenseDeep, navigate your browser to: https://app.sensedeep.com.

To learn more about SenseDeep please see: https://www.sensedeep.com/product.

Please let us know what you think, we thrive on feedback. dev@sensedeep.com.

SenseDeep DynamoDB Studio

Michael O'Brien — Fri, 29 Oct 2021 06:20:18 +0000

Introducing the SenseDeep DynamoDB Studio.

SenseDeep has a complete DynamoDB developer studio to support your DynamoDB designs and single-table development.

It includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics.

SenseDeep DynamoDB studio is a comprehensive set of DynamoDB tools that are single-table "aware". This means SenseDeep can understand your single-table designs and application entity data and can guide your design, queries and monitoring based on this deeper understanding.

Background

DynamoDB best practices are evolving quickly as developers are realizing how to exploit the power behind the deceptively simple DynamoDB design. Designs with single-table design patterns, key overriding and composition, sparse indexes, query optimization and more powerful single-table access libraries such as OneTable are becoming common place.

However, managing single-table data and performance can often feel like you are peering at Assembly Language. Packing disparate data items into a single-table can make navigating, organizing and viewing data difficult. Furthermore, single-table design techniques such as prefixed and mapped attribute names exacerbate this problem and can make interpreting keys tough.

New tools are needed that understand the single-table schema and its relationships. These tools should support schema creating and be able to present and organize your data logically according to your application entities.

Data Browser

The SenseDeep data browser can query, manage and modify your table data. It supports browsing by scan, query or by single-table entities.

While the data browser can be used to browse and manage any DynamoDB table, it is turbo-charged when a schema describing the data is imported (such as from a OneTable schema) or defined using the SenseDeep single-table designer. When using a data schema, SenseDeep is able to "understand" your data and organize and present data items as application entities instead of raw, encoded DynamoDB items.

The SenseDeep data browser intelligently displays and formats data according to your single-table schema. This not only makes your encoded single-table data easier to understand, but it guides and validates your changes and prevents schema-breaking and application breaking data changes.

Queries

You can perform DynamoDB scans, native queries or queries by application entity.

When scanning, you select the index to scan and provide optional additional filtering attributes. Attributes can be combined in an AND or OR expression.

When querying, you specify the index and partition key with optional sort key value. Additional filters may also be specified.

When querying by Entity, you specify the index and application entity model name that is defined in the schema. SenseDeep then intelligently prompts you for the required attributes that comprise the keys to retrieve the data items.

Saving Queries

Queries may be saved to your database table where they are persisted in the schema. You can load and delete queries using the Queries button.

Data Visualization

SenseDeep groups, organizes and color-codes query results for maximum clarity.

Cells are color coded:

A cell is green if it can be edited. Click on the cell to modify the value.
A cell is pink if it has its value derived from a schema value template.
A cell has a wavy gray background if it is not relevant for this item (as defined by the schema).
A cell has a blue underline hot-link if it is a reference to another item and can be click to jump to the referenced item.

Hot Linked Items

If you control-click (or Cmd-Click) on a hot-link, you can quickly traverse related items. SenseDeep determines the relevant query to locate the item and automatically fills the index, key and filter attributes accordingly. You can save these queries using Queries -> Save. Queries are entered into your browser history so you can click the browser Back button to easily jump backwards to the original query.

Show Templates

A useful single-table design technique is to compose key values using templates that combine the values of other attributes at runtime to calculate the key values. It is often useful to view the value templates vs the calculated key values. The Templates toggle above the table switches between displaying the template values vs the actual data values.

Similarly, there may be some attributes that are designated as hidden. Changing the hidden toggle will display or hide these attributes.

Context Menu

On data cells, you can right click to display the context menu. The options are:

Copy to copy the current item to the clipboard in JSON format.
Design Schema will jump to the SenseDeep single-table designer for the entity on which the current item is based.
Edit will open the editor slide-out panel for easy editing of the whole item.
Follow Reference behaves the same as Cmd-Click to follow a hot link reference.
Generate Mock Data option can be used when developing to generate sample data, such as email addresses or phone numbers.

Modifying Data

You can modify data inline by clicking on any green cell. Once modified, the Save button will be displayed above the items to persist the changes to the table.

To add a new item, click the Add Item button. You can select the desired entity model (if a schema is present) and it will intelligently prompt you for the appropriate attributes.

You can also edit by clicking on the Edit pencil icon at the start of each item. This will display a slide out editor panel that will display only the attributes of the item, organized vertically. Click on a green cell to modify the contents. The red cells have their value derived from other attributes and cannot be edited. You can modify the value template in the schema via the SenseDeep single-table designer.

When you click save, the changes are accepted, but you still must click the Save button on the query page to persist results to the table.

Import / Export

No man is an island and your data must be easy to export or import. SenseDeep provides and Export to AWS Workbench option. This will export your schema and data items into a WorkBench model. This model can also be imported by the Dynobase tool.

When exporting, you should limit the amount of data you export. WorkBench models are for development and are designed for limited data sets. You can also export in a JSON backup format.

When importing, you can import a model to an existing (empty) table, or you can dynamically create a new table to hold the imported model.

Single Table Designer

The SenseDeep Single Table Designer provides an easy, intuitive interface to create, manage and modify your single-table schemas.

Single-table schemas control how your application data are stored in DynamoDB tables. Schemas define your application entities, table indexes and key parameters. Via schemas, complex mapped table items can be more clearly and reliably accessed, modified and presented.

The designer stores your single-table schemas in your table. In this manner, your table is self-describing as to how table data should be interpreted. You can export schemas and generate JSON, JavaScript or TypeScript data/code to import into your apps.

The SenseDeep single-table designer works best with the OneTable library, but should work with any consistent single-table design.

Naming and Versioning

Schemas are named and versioned. The default schema is called the Current schema. Other schemas can have any name of your choosing. For example "Prototype".

Schemas have version numbers so that data migrations can utilize the correct versioned schema when upgrading or downgrading data items in your table. The migration manager will select and apply the correct versioned schema when data migrations are run.

Version numbers use Semantic Versioning to indicate and control data compatibility.

Entity Models

An entity model defines the valid set of attributes for an application entity. Each attribute has a defined name, type and a set of properties that may be utilized by your DynamoDB access library such as OneTable or an ORM of your choosing.

From the list of entity models, you can click on an entity to modify, click Add Model to add a new entity model.

Via the Edit Schema button you can modify the schema name or version. If the schema is a non-current schema, you can also click Apply to Current to apply the contents of the displayed schema to the saved Current schema in the table. This overwrites the previous Current schema.

Entity Fields

Clicking on an entity attribute will display a slide out panel to edit the properties of that attribute.

You can modify the type and any other properties of the type. These properties are defined in the schema and may (or may not) be implemented by your DynamoDB access library. All these are implemented by OneTable.

The properties include:

Name — the name of the attribute.
Default Value — a default value to set if not defined when created.
Enumerated values — a list of possible values for the attribute.
Encrypt — hint the attribute should have an extra layer of encryption.
Filter — indicates the attribute can be used in filter expressions.
Hidden — the attribute should be hidden in query results by default.
Mapped Name — a shorter real attribute name when storing in the table.
Nulls — indicates that nulls should be stored in the table vs being removed.
Required — the attribute is required to be present when created.
Data Type — the attribute data type.
Validation Expression — a regular expression to validate all writes to the table.
Value template — a JavaScript value template to compose the attribute based on other attribute values.
Unique — indicates the attribute must always hold a unique value over all such items.

Export

You can also export your schema in JavaScript, TypeScript or JSON formats via the Export Schema button from the Models list. You can utilize the exported file directly in your OneTable apps to control your database interactions.

Provisioning Planner

The DynamoDB Provisioning page displays your current and projected provisioning, utilization and costs.

While there are various DynamoDB calculators, they are "theoretical" and not based on actual data usage. The SenseDeep provisioning planner displays displays the actual costs of your current billing plan based on real, live data. It then compares this usage with the the cost of an alternate plan if you were to switch your billing plan from OnDemand to Provisioned or vice-versa.

Database Migration Manager

The SenseDeep database migration manager provide a controlled way to upgrade or downgrade your table data.

It displays what migrations have been applied to your data and what migrations are outstanding. It shows what is the migration version for your current data and schema. You can upgrade by running outstanding migrations or downgrade by reversing migrations.

SenseDeep can orchestrate DynamoDB migrations controlled by the OneTable Migration library. You can use this library even if you are not using OneTable for your apps. Check out the OneTable Controller sample OneTable Controller in GitHub and deploy to host and manage your data migrations.

Read Configuring OneTable Migrate for more information and setup.

Table Metrics

SenseDeep provides both standard AWS DynamoDB metrics and enhanced single-table metrics.

The DynamoDB standard metrics displays overview metrics and operation details for a whole DynamoDB table. These metrics come from the standard AWS CloudWatch DynamoDB metrics.

DynamoDB Single-table designs are more complex and require per entity/model performance monitoring and metrics. Traditional monitoring covers table level and operation level metrics only. What is missing is the ability to see single-table entities and their performance and load on the database.

SenseDeep provides single-table metrics for apps using DynamoDB OneTable or for any JavaScript app that utilizes DynamoDB Metrics.

Table Management

SenseDeep can manage the DynamoDB tables for your enabled clouds. You can create and delete tables and indexes for your tables.

SenseDeep will automatically discover your tables and will dynamically update this list as new tables are created or destroyed. The table list includes the table size, number of items, billing scheme and provisioned capacity.

The SenseDeep table management is not meant to replace appropriate "infrastructure-as-code" deployment of tables to production. Rather, it intends to provide a quick and easy way to create and manage tables and indexes while developing your DynamoDB applications.

Summary

Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.

More?

Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App or learn more at https://www.sensedeep.com.

About SenseDeep

SenseDeep is an observability platform for AWS developers to accelerate the delivery and maintenance of serverless applications.

To try SenseDeep, navigate your browser to: https://app.sensedeep.com.

To learn more about SenseDeep please see: https://www.sensedeep.com/product.

Please let us know what you think, we thrive on feedback. dev@sensedeep.com.

Understanding your DynamoDB Single Table Performance

Michael O'Brien — Thu, 16 Sep 2021 07:54:03 +0000

Best practices for DynamoDB have evolved to favor single-table design patterns where one database table serves the entire application and holds multiple different application entities.

This design pattern offers greater performance by reducing the number of requests required to retrieve information and lowers operational overhead. It also greatly simplifies the changing and evolving of your DynamoDB designs by uncoupling the entity key fields and attributes from the physical table structure.

However, managing single-table data and performance can often feel like you are peering at Assembly Language. Composite keys with prefixed and mapped attribute names are single-table design techniques but they can make just reading a single-table item quite difficult.

What is needed are tools that "understand" the single-table schema and can present and organize your data logically according to your application entities.

To meet this need we've created the DynamoDB Metrics library which calculates and emits detailed single-table performance metrics for DynamoDB.

This post looks at our libraries DynamoDB Metrics, OneTable and the SenseDeep platform that understand your single-table design schema and can create and present detailed metrics to graphically show how your single-table designs are performing.

Single Table Monitoring

So what are the kinds of questions that DynamoDB Metrics can answer?

Which single-table entity/model is causing the most load and is consuming the most RCU or WCU?
Which customer tenant is causing the most load and how much should they be billed?
Which app or function is causing what percentage of load on DynamoDB and is consuming the most RCU or WCU?
Which queries are the most inefficient (items vs scanned) and by which app or model?
Which operations are being used the most?
Which entity is using performing scans or other operations?

These questions and others can be answered by using detailed metrics for DynamoDB that profile performance at an application entity/model level.

DynamoDB Metrics Features

The DynamoDB Metrics library is an NPM module for Node applications that captures and emits detailed DynamoDB metrics. It has the following features:

Creates detailed CloudWatch metrics for Tables, Indexes, Apps/Functions, Entities and DynamoDB operations.
Emits metrics using CloudWatch EMF for zero-latency metric creation.
Supports AWS V2 and V3 SDKs.
Simple easy integration.
Very low CPU and memory impact.
Clean, readable small code base (<400 lines).
Full TypeScript support.
No dependencies.
Optionally integrates with SenseLogs the Serverless Logger for dynamic control of metrics.
Supported by the free SenseDeep Developer Plan for graphical DynamoDB single-table monitoring.

Metrics Galore

DynamoDB Metrics captures detailed statistics across 5 dimensions:

Table — Per table metrics
Tenant — Per tenant metrics
Source — Per application, module or function identification
Index — Primary or global secondary index
Model — Application single-table entity / model name
Operation — DynamoDB low-level operation: GetItem, PutItem, etc.

You can drill-down to see metrics aggregated by table, tenant, source, index, model or operation. This enables you to pin-point exactly where performance issues may be lurking.

For each of these dimension combinations, DynamoDB Metrics emits the following metrics:

read — Read capacity units consumed
write — Write capacity units consumed
latency — Aggregated request latency in milliseconds
count — Count of items returned
scanned — Number of items scanned
requests — Number of API requests issued

With these metrics, you can see precisely who is consuming read and write capacity, which requests are running long, which requests are inefficient and are scanning the table.

How to get DynamoDB Metrics

There are two ways to get these wonderful single-table metrics for DynamoDB.

The DynamoDB Metrics NPM library can be used by any Node application using DynamoDB. It is configured as AWS SDK middleware and efficiently captures request details with minimal overhead.
Alternatively, you can use the OneTable library that has this support built-in and get all the other benefits of OneTable.

DynamoDB Metrics

To configure DynamoDB Metrics, load the library and pass your DynamoDB client instance to the Metrics constructor. The other parameters tell Metrics how to understand your index and key structure.

import Metrics from 'dynamodb-metrics'
const metrics = new Metrics({
    client: client,
    indexes: {primary: {hash: 'pk', sort: 'sk'}},
    separator: '#',
})

You can read more about how to configure Metrics at DynamoDB Metrics README.

OneTable Support

To enable DynamoDB Metrics using OneTable, just add metrics to your OneTable constructor and specify the name of your application or Lambda function.

const table = new Table({
    metrics: {source: 'acme:launcher'}
})

OneTable uses your defined OneTable schema to understand your key structure.

How to View DynamoDB Single Table Metrics

You can view DynamoDB single-table metrics using CloudWatch or the SenseDeep Serverless Platform.

Viewing via SenseDeep

SenseDeep has pre-configured dashboards and graphs to assist in visualizing your DynamoDB metrics. You can drill down and view metrics at the table, tenant, source, index, model or operation dimension level for any desired time period.

It is easy to see which application or function is consuming read/write capacity and how your app data entities are using DynamoDB.

SenseDeep also provides intuitive capacity planning and provisioning assistance.

CloudWatch

Using CloudWatch Metrics, you can see cards for the DynamoDB Metrics dimension combinations.

The metrics are created under the SingleTable/Metrics.1 namespace. Clicking on a card provides a list of dimension combinations to graph.

Under the Hood

The DynamoDB Metrics library emits metrics using the CloudWatch EMF log-based metrics format. This permits zero-latency creation of metrics without impacting the performance of your Lambdas. EMF allows metrics to be emitted without blocking as would be the case with a normal blocking API.

DynamoDB Metrics will only emit metrics for dimension combinations that are active. If you have many application entities and indexes, you may end up with a large number of metrics. If your site uses all these dimensions actively, your CloudWatch Metric costs may be high. You will be charged by AWS CloudWatch for the total number of metrics that are active each hour at the rate of $0.30 cents per hour.

If your CloudWatch costs are too high, you can minimize your charges by reducing the number of dimensions. The dimensions emitted can be modified via the dimensions constructor property. Alternatively, you can dynamically enable and disable metrics via the LOG_FILTER parameter.

DynamoDB Metrics are buffered and aggregated to minimize the load on your system. If a Lambda function is reclaimed by AWS Lambda, there may be a few metric requests that are not emitted before the function is reclaimed. This should be a very small percentage and should not significantly impact the quality of the metrics. You can control this buffering via the Metrics constructor.

Summary

Gaining insight into single-table design patterns is the new frontier. Previously, single-table designs with DynamoDB has been a black box and it has been difficult to peer inside and see how the components of your design are operating.

DynamoDB Metrics provides an easy way to instrument your code and gain these insights. SenseDeep provides a free developer plan so you can view and analyze these metrics with graphical dashboards.

More?

Download DynamoDB Metrics from NPM at DynamoDB Metrics

For the most elegant way to create single-table designs, consider OneTable.

And you can get a free developer license for SenseDeep at SenseDeep App or learn more at https://www.sensedeep.com.

About SenseDeep

SenseDeep is an observability platform for AWS developers to accelerate the delivery and maintenance of serverless applications.

To try SenseDeep, navigate your browser to: https://app.sensedeep.com.

To learn more about SenseDeep please see: https://www.sensedeep.com/product.

Please let us know what you think, we thrive on feedback. dev@sensedeep.com.

Dynamic Log Control for Serverless

Michael O'Brien — Thu, 19 Aug 2021 22:48:12 +0000

Serverless apps are an immensely powerful way to build highly scalable and available services.
But, serverless apps are also uniquely difficult to monitor, debug and manage due to their distributed components, stateless nature, short execution lifespans and limited access to configuration.

Observability has been popularized as the solution to this dilemma by "instrumenting your functions verbosely" and "wrapping all service boundaries with copious logging".

However, this can degrade critical production performance and send logging and metric costs spiraling.

So what is the solution?

This post explores a simple, yet effective solution via Dynamic Log Control.

Observability

Distributed serverless systems need to be easily monitored for current and future failures. To achieve this, such systems need to be "Observable".

Observability aims to provide granular insights into the behavior of a system with rich context to diagnose current errors and anticipate potential failures. Observable systems integrate telemetry to monitor the system via logs, metrics, events and traces.

Easy you say, just add lots of logging and metrics.

However the painful tradeoff with observability is the hard cost of the telemetry vs the potential benefit of that observability in the future. Each line of logging costs in terms of CPU and log storage. Copious logging can lead to rude logging cost surprises.

What is needed is a way to minimize the cost of telemetry until needed and to scale it up and down according to the need.

Of course, you need a baseline level of logging and metrics so you can have an accurate assessment of your services. But you often need greater insight in response to certain alarms or triggers.

Dynamic Log Control

Serverless apps pose challenges for the dynamic scaling up and down of logging.

Serverless functions are ephemeral. They are not like an EC2 server or long lived docker applications. They often last only milliseconds before terminating. They keep little or no state and fetching state from a database may be too great an overhead to incur on every function invocation.

However, there is a technique that can be adapted for dynamic log control that is well proven and understood: Environment Variables.

Environment Control

When AWS Lambda loads a function, it provides a set of environment variables that can be read by the function with zero delay. Via these variables, you can provide log control instructions to vary the amount and focus of your logging.

Here is the AWS console's Lambda environment configuration page.

If you change the environment variables (without redeploying or modifying your function), your next invocation will run with the new environment values.

So using special LOG environment variables, we can communicate our desired log level, filtering and sampling, and have our functions respond to these settings without modifying code or redeploying.
Sweet!

What are the downsides: Changing environment variables will incur a cold-start the next time the functions are invoked, but that is typically a short, one-off delay.

Log Control

To control our logging, we propose three log environment variables:

LOG_FILTER
LOG_OVERRIDE
LOG_SAMPLE

Depending on your log library (SenseLogs, Pino, Winston, ...) you specify your log level or channel via these variables.

The LOG_FILTER defines the set of enabled log levels or channels. Enabled levels will emit output. Disabled levels will be silent.
For example:

LOG_FILTER='error,warn'

This will enable logging output in your Lambdas for log.error() and log.warn() but disable output for calls to trace or debug.

The LOG_OVERRIDE will replace the LOG_FILTER for a given period of time before reverting to the LOG_FILTER settings. LOG_OVERRIDE is prefixed with an expiry time expressed as a Unix epoch in milliseconds since Jan 1 1970.

For example:

LOG_OVERRIDE='1629806612164:debug,trace'

The LOG_SAMPLE is a sampling filter that applies to a given percentage of requests. The list of levels is is prefixed with a percentage.

For example, this will trace 5% of requests.

LOG_SAMPLE='5%:trace'

Logging Libraries

This technique will work with Node, Python and Java Lambda functions. It is easy to configure with most popular log libraries including: Bunyan, Debug, Pino, SenseLogs or Winston.

An ideal logging library will permit custom levels so that latent logging code can be embedded in your function without incurring a run-time overhead for most invocations. When required, you can add that custom level to your LOG_FILTER or LOG_OVERRIDE and turn on that custom log output. Custom levels should be able to be enabled and disabled without impacting other levels.

It is very important that your logging library has extremely low overhead for disabled log levels. Otherwise, the benefit of dynamically scaling is lost.

Implementations

I've included below some examples for NodeJS using the SenseLogs and Pino logging libraries.

To see code samples for Bunyan, Debug, Winston or Python please checkout our Dynamic Logging Samples which has detailed code samples for each library and Python using these techniques.

Also available via GitHub gists at: Dynamic Log Control Gist

SenseLogs
Pino

Controlling SenseLogs

Here is a sample of how to use this technique with SenseLogs which has builtin support to respond to the LOG_FILTER, LOG_OVERRIDE and LOG_SAMPLE environment variables. SenseLogs is an extremely fast serverless logger that supports custom log channels. It has almost zero cost for disabled log levels.

import SenseLogs from 'senselogs'

let log = new SenseLogs()

exports.handler = async (event, context) => {
    //  This will be emitted if LOG_FILTER contains 'debug' as a log level
    log.debug('Hello world')

    //  EMF metrics can also be dynamically controlled
    log.metrics('trace', 'AcmeRockets/CriticalLaunchFailure', {explosion: 1})
}

Learn more at Serverless Logging with SenseLogs.

Controlling Pino

Here is a sample of how to use the popular Pino general purpose logger.

This sample uses a small amount of code to parse the LOG environment variables and setup the Pino logger.

import Pino from 'pino'

let {LOG_FILTER, LOG_OVERRIDE, LOG_SAMPLE} = process.env

if (LOG_OVERRIDE != null) {
    let [expire, level] = LOG_OVERRIDE.split(':')
    if (level && Date.now() < expire) {
        LOG_FILTER = level
    }
}
let sample = 0, sampleRate, sampleLevels
if (LOG_SAMPLE != null) {
    [sampleRate, sampleLevel]s = LOG_SAMPLE.split(':')
}

const pino = Pino({name: 'pino', level: LOG_FILTER})

exports.handler = async (event, context) => {
    if (sampleRate && (sample++ % (100 / sampleRate)) == 0) {
        //  Apply the sample levels
        pino.level = sampleLevel
    }
    pino.debug('Debug message')
}

Managing Log Configuration

When you have an important issue to diagnose, you can now scale up your logging by setting the LOG_OVERRIDE environment variable to increase the log level for a period of time. Depending on your log library, you may also be able to focus your logging on a specific module by using custom levels.

You can modify your environment configuration via API, the AWS Console, the AWS SDK or using the SenseDeep Developer Studio.

AWS CLI

The AWS CLI has support to update the function configuration using update-function-configuration.

For example:

EXPIRES=$((($(date +%s) + 3600) * 1000))
aws lambda update-function-configuration --function-name MyFunction \
    --environment "Variables={LOG_OVERRIDE=${EXPIRES}:error,info,debug,trace,commerce}"

This calculates an expiry time and overrides the log levels for one hour to include the error, info, debug and trace levels and the custom commerce level.

AWS Console

You can also change these environment values via the AWS console.

Simply navigate to your function, select Configuration then Environment Variables then Edit and modify then save.

SenseDeep Serverless Studio

Better still, the SenseDeep Serverless Developer Studio provides an integrated, high-level way to manage these filter settings.

Summary

Using this technique, you can easily and quickly scale up and down and focus your logging as your needs vary. You can keep a good base level of logging and metrics without breaking the bank, and then increase logging on-demand.

More?

The logging library code samples are available at Log Control Samples.

SenseLogs is available from GitHub SenseLogs or NPM SenseLogs.

And you can get a free developer license for SenseDeep at SenseDeep App or learn more at https://www.sensedeep.com.

<@ partial('about') @>