Web APIs and Batch Jobs Communicate Failure Differently

Long Story Short

• Web APIs can communicate failure through HTTP status codes, but batch jobs need to terminate in a way that allows the execution environment to recognize that the process has failed.
• In this case, I changed the error handling to throw an exception. This caused failed Lambda invocations to be reflected in the CloudWatch Errors metric, which allowed the alarm to trigger.
• Error handling does not end inside the application. We also need to think about whether failures can be correctly observed by the monitoring and alerting systems outside it.

Context

Recently, I had the opportunity to implement a mechanism that sends a Slack notification when a batch job running on AWS fails.

The batch job itself was an existing Lambda function that was already running in production, so I initially thought the task would be relatively simple.

The setup was to monitor Lambda execution errors with CloudWatch and send notifications to Slack through AWS notification services.

The AWS-side configuration itself was not particularly complicated.

However, when I looked at the existing batch code that we wanted to monitor, I found a problem.

An Error Was Happening, but the Job Wasn't Failing

The existing error handling included code like this:

return false

or:

return { status: 500 }

The intention was probably to indicate that an error had occurred.

From Lambda's perspective, however, that is not what happens.

Returning false or { status: 500 } simply means that the Lambda function returned a value and completed successfully.

Just because the returned value contains the number 500 does not mean that Lambda automatically treats the invocation as a failure.

As a result, even though the application code intended to handle the situation as an error, the execution environment still saw it as a successful invocation.

Because of that, the invocation was not treated as a function error and was not reflected in CloudWatch's Errors metric.

The CloudWatch Alarm therefore did not trigger, and no Slack notification was sent.

Building the monitoring mechanism made me realize that the existing error handling was not communicating the failure correctly to the systems outside the application.

I Couldn't Apply the Same Mental Model as a Web API

What I found interesting was the difference between how Web APIs and batch jobs communicate failure.

In a Web API, we use HTTP status codes to communicate the result of a request to the client.

For example:

HTTP 500 Internal Server Error

By returning an HTTP 500 response, the client can recognize that the request has failed.

When you spend a lot of time working with Web APIs, it becomes natural to think:

If something goes wrong, return 500.

But what I was working with this time was not a Web API responding to an HTTP request. It was a batch job running on Lambda.

There was no client waiting for an HTTP response.

What mattered was not returning a value that contained 500, but making the Lambda invocation itself fail.

In this case, I refactored the code to throw an exception when an error occurred:

throw new Error(...)

This caused the Lambda invocation itself to fail, which was then reflected in the CloudWatch Errors metric.

Error Handling Doesn't End Inside the Application

When I started this task, I thought I was simply adding a Slack notification for batch failures.

But once I followed the notification flow back to the existing code, I ended up asking a more fundamental question:

Is this application actually communicating its failures correctly to the outside world?

Writing an error log.

Returning false.

Returning a value containing 500.

If you only look at the application code, all of these can appear to be forms of error handling.

But if the Lambda runtime and CloudWatch cannot observe the execution as a failure, that error handling does not connect to monitoring or alerting.

This experience made me realize that when thinking about error handling, I should consider not only:

"How is the error handled inside the code?"

but also:

"Who needs to observe this failure, and how will they know that it happened?"

Conclusion

Web APIs communicate success and failure to clients through HTTP status codes.

Batch jobs are different. We need to think about how the execution environment determines whether a job succeeded or failed.

Implementing this Slack notification mechanism made me realize that the familiar Web API mindset of "return 500 when something fails" cannot simply be applied to batch jobs.

Error handling is not only an application-level concern.

I learned that part of error handling is also making sure that a failure can be correctly observed by the execution environment and the monitoring systems around the application.