Previous: Sensitive Data Encryption
The Signup Service creates the user and profile records, and the User Profiles Service encrypts protected data before it is stored. There is still one task left after successful registration: notifying the user.
That work is handled by the Event Processor.
The Event Processor provides a controlled way for services to publish events, route those events through configurable handlers, and deliver the resulting messages through external channels such as email, REST APIs and messengers.
The signup email is one example. The same mechanism can support many kinds of follow-up work without placing that work directly inside the service that created the original event.
Why use an Event Processor?
A business workflow should not need to perform every follow-up operation synchronously.
After a user signs up, the Signup Service needs to return the new user and profile identifiers. It should not also need to know how to select an email template, configure an email provider, send the email, handle delivery failure, or retry a temporary error.
Instead, it publishes a signup event.
The service that produces an event and the service that consumes it remain independent. Kafka topics provide the boundary between them: producers publish events to a topic, while consumers subscribe to and process events from that topic. Kafka documentation describes producers and consumers as decoupled, with events organized and durably stored in topics.
Event ingress
The Event API receives events through:
POST /event/send/{event_topic}
The endpoint validates the caller’s permissions, verifies that the requested topic is allowed, adds service-level context, and publishes the event to Kafka.
Each event message contains:
event_id
project_id
data
token, when provided
The Event API generates a UUID event_id for every incoming event. This provides a stable identifier that can be used to trace the event through processing stages and logs.
The project_id identifies the project context. When the caller supplies an authorization token, the Event API includes it in the Kafka message so downstream actions can use the required authorization context.
The API prevents callers from publishing directly to internal service topics. It also verifies that the requested Kafka topic exists. If the requested topic is missing from the cached topic list, the service refreshes the list before returning a not-found error.
Shared Kafka producer
The Event API creates one shared AIOKafkaProducer during the FastAPI application lifespan.
At startup, the service:
- Initializes its shared HTT P client
- Initializes logging and authorization
- Loads the available Kafka topics
- Creates and starts a Kafka producer using the configured security settings
FastAPI lifespan handlers are designed for logic that runs before the application begins serving requests and during shutdown, which makes them suitable for managing shared resources such as external clients and message producers. AIOKafkaProducer.send_and_wait() publishes a record and waits for the send operation to complete.
Using one producer for the application avoids creating and stopping a separate Kafka producer for every incoming event.
EVENT, PREPARE, and PUSH
The intended processing model has three phases:
EVENT → PREPARE → PUSH
EVENT
The Event API receives an event from another service, validates it, enriches it with event and project context, and publishes it to Kafka.
The Event API does not need to know whether the event will lead to an email, an external REST call, a messenger notification, or another action. It only accepts and publishes the event.
PREPARE
Some events need transformation before they can be delivered. An email must be rendered from a template, a REST endpoint may require a particular JSON structure, and a messenger text may require its own formatting.
A configured processor consumes an event from one topic, runs one or more preparation actions, and publishes the resulting delivery packages to the next topic.
For example:
{
"topic": "accrual-send",
"actions": [
{
"module": "prepare_json",
"next_topic": "RestAPI"
}
],
"description": "awarding points",
"object_type": "event.processor"
}
In this example, the processor receives an event from the accrual-send topic, invokes the prepare_json module, and sends the prepared result to the RestAPI topic.
PUSH
The PUSH phase delivers the prepared package to an external destination.
A transport action can use protocols such as:
- SMTP
- REST
- Messenger
- Other transports added later
For example:
{
"topic": "accruals",
"actions": [
{
"module": "restapi_send",
"next_topic": "accrual-send",
"template_name": "accrual"
}
],
"descriptions": "retrieving user points for payment processing",
"object_type": "event.transport"
}
This handler consumes messages from the configured topic, invokes the restapi_send module, and can use the configured accrual template when preparing the destination request.
The Event Processor keeps its own internal headers separate from headers required by an external destination. This matters because a third-party REST API may require its own authorization or protocol-specific headers that should not be confused with service-to-service context.
Configurable handlers and actions
The Event Processor is configuration-driven.
Its configuration API manages:
- Event-handler configurations
- Email-sender configurations
- Message templates
- Access-token configurations
These records are stored in the configuration registry. They are loaded as active records and can be scoped by project.
Configurations allow routes, actions, templates, delivery settings, and access details to be managed outside the core event-ingress endpoint.
For example, templates can be loaded by project and template name. Email configuration can be retrieved separately from event-handler configuration. Access configuration can be stored per project rather than embedded in service code.
Handler lifecycle
At startup, the service loads active configurations from the configuration registry.
The current handler configuration types include:
event.processor
event.transport
event.handler
Each configuration creates an event handler with:
- A Kafka topic to consume
- A handler type
- A Kafka consumer for that topic
- Access to the application-level producer
- A list of actions loaded from configuration
- A directory containing action modules
Each action identifies a module that implements its logic and a next_topic for the action result.
When a handler consumes an event, it runs the configured actions asynchronously. Action tasks are named with the event ID, handler topic, and action name, making concurrent work easier to trace during logging and debugging.
Actions that return a result publish a new event to their configured downstream topic, where another handler can continue processing.
Incoming event
→ consume from topic
→ run configured actions
→ collect results
→ publish results to next topics
This allows processing to be expressed as a pipeline of independently configured stages rather than as one large synchronous workflow.
Configuration loading
Configuration records are loaded from the configuration registry through asynchronous HTTP requests.
The loaders select only active records. They filter by object_type and optionally by project_id. This allows multiple projects to use the same Event Processor while maintaining separate handler configurations, templates, email settings, and access configurations.
The current API supports creating and updating event configurations. A planned improvement is to notify the event-handler container when a configuration changes so the relevant handler can start or restart without a full service restart.
Failure handling
The intended model treats delivery outcomes as events.
After a PUSH operation, the system can produce outcomes such as:
_success
_fail
A more flexible configuration can specify which event should be emitted after successful or failed delivery.
Failure behavior can also depend on an external response, such as an HTTP status code or an application-specific error code like USER_NOT_FOUND.
Potential configured responses include:
RETRY
STOP
A retry policy can specify a delay before another delivery attempt. A stop policy ends processing for that delivery.
These capabilities describe the intended direction of the Event Processor. They should be understood as planned behavior until the associated retry, result-event, and failure-routing logic is implemented and verified.
Signup email example
The signup workflow ends by sending an event after user and profile creation succeeds.
The Signup Service does not send the email directly. It publishes an event containing the information required by the configured event-processing pipeline.
The Event Processor can then:
- Receive the event through the Event API
- Publish it to Kafka
- Consume it through a configured handler
- Prepare an email package using an action and template
- Send the email through a configured delivery mechanism
- Optionally publish a success or failure event
The registration workflow can complete independently of delivery, while notification behavior remains configurable.
Closing perspective
The Event Processor is built around a simple principle: applications publish facts about what happened, while configurable handlers decide what should happen next.
The Event API controls event ingress. Kafka carries events between stages. The configuration registry stores handler rules, templates, delivery settings, and access details. Action modules prepare or deliver messages.
The architecture is still evolving. The shared application-level Kafka producer is already in place, while dynamic handler reloads, configurable retries, and success or failure events are planned improvements.
In the next article, I will discuss the service-to-service security model: tokens, authentication, authorization, and permissions.
Top comments (0)