Building Logging V2 for Ticketon

~15 minutes read time

When being a Discord server admin/manager or moderator, it is important to know what happens in your server. Discord provides a basic Audit Log, which keeps track of the most important events like moderation actions, server/channel updates, role changes, and more.

This can be improved by using a bot like Sapphire (#NotAnAd) to log more events directly as messages in a channel of your choice, instead of having to go in the server settings and check the Audit Log - which is also limited to the last 90 days of events (or was it 30? I don't remember).

However, many bots don't have logging features which track what happens to their own configs and actions - and if, then just stuff they do like moderation actions. But I can't count the number of times I had to answer questions like "How can I know who misconfigured the bot?" in the Sapphire support server. (One might think that I'm a fan of Sapphire)

The first logging system

The first MVP of the Bot had a very basic logging system where I had an AuditLogEntry mongoose model which was used to manually create documents when a ticket category was created/updated/deleted, a ticket created/closed/deleted, a custom message created/updated/deleted, and a few other events. This was enough to get started, but it was very limited and not very flexible.

For the MVP I also decided to just have an audit log page in the web dashboard where logs could be explored. The UI was also very ugly - mostly displaying the raw JSON data of the log entries. But it was enough to get started and to have a basic logging system in place.

However, it was very messy as there would be two different places where logs could be generated: the web dashboard and the bot itself. For configuration changes, the web dashboard would need to compare the old and new documents, diff the fields - including nested fields - and generate a log entry. The result was a lot of code duplication and a lot of manually juggling the fields for every data model.

The new idea

From the beginning I wanted to have a logging system that would also send logs to a Discord channel, so that server admins/mods could see what happens in their server without having to go to the web dashboard. This was also requested in December 2025 by a user in the support server in the #suggestions channel.

I then decided in early July 2026 to start working on a new logging system to unify the logging system and make it more flexible and easier to use. The idea was to have a single logging system that would handle all logs, from both the bot and from the web dashboard, and to have a single place where logs could be generated and sent to a Discord channel. This lead to a general architecture idea:

Now I asked myself, how do I make this work? Which technologies do I use? How do I make it scalable and maybe even be standalone from the bot and the web dashboard?

My first idea was to have a Worker (a Cloudflare Worker to be exact) that would have an API endpoint where the bot and the web dashboard could send log entries. It would also have a D1 database where all logs would be stored with a UUID (v7) primary key. The Worker would then have put logs in a Queue (Cloudflare Queue) as they come in and then process them sequentially, sending them to the Discord channel and storing them in the database.

The requests to the worker would be secured by signing them with a secret key, and the worker would verify the signature before processing the request. This would ensure that only the bot and the web dashboard could send logs to the worker. This is a very common pattern for webhooks and is used by a number of my projects.

Since I don't want the bot token to be in more places than necessary, I thought it would be best to have the worker send the logs to a Discord webhook instead of using the bot token. This would also allow for better ratelimit handling, as webhooks fall under a different ratelimit than bot tokens. The worker would then have a configuration for the webhook URL and the channel ID where logs should be sent.

This issue that emerged from this idea was - how would the worker get the configuration for the webhook URL and the channel ID? The bot must do this and the worker must somehow access it. There were only two possible solutions: either the bot would have an API endpoint where the worker could request the webhook config or the worker would need to create it.

However, I wanted to have a special feature which I saw in the Sapphire bot: Webhooks are created on demand when they are needed. When there is an existing webhook, it would just reuse it. This would result in tons of requests to the bot to get or create webhooks, which would be a lot of overhead and would also require the bot to be online all the time.

GDPR compliance was also a concern. The bot already does this in a regular cleanup process - the worker would have to clean up after its logs by its own. This would also either require the worker to request even more data from the bot or the bot sending the worker the data when something needs to be done.

Phew. That was... quite a thought. This was so much to think about it was tiring me a lot. I decided to take a break and go to sleep. I would think about it tomorrow. I went to bed and had a good sleep.

As I got up the next morning, I had a clear head and it suddenly all fell into place. My goal was unification - A single writer/schema, kill the duplicated AuditLog Mongoose model between the bot and web repo. Even though Cloudflare Workers is designed to scale, when there are hundreds of log events per minute, the sheer amount of requests between the bot, the worker and Discord would be so much overhead that it would mainly be a waste of resources.

Then I tried a new approach. What if I just... not used Cloudflare Workers? Maybe integrate it (sort of) into the bot itself?

What I learned until now was: When you're building something, you should always set a goal and a vision for it. Sometimes take a step back and think about the bigger picture. Don't get lost in the details and don't be afraid to change your approach if you find a better one.

A different approach

Something else that was important was logging configs. After some discussing with our best friend Claude, I decided to also integrate the logging configs into the logging system in the bot - and not the dashboard.

The board infrastructure would look like this:

This architecture is just the backend part.

There are basically two possible emit sites: The bot and the dashboard.

The bot would just call log() to start a log entry. The dashboard backend would also call a log() function to start a log entry - this one just sends it over to the bot via a REST API endpoint there (of which the bot already has a few). The Log Manager, the so-called dispatcher would then handle the log event, save it in the DB and enqueue it to the Log Queue. The Log Queue would then process the log entry and send it to the Discord channel using the webhook-on-demand thing.

The log entry would be created in the database even before it is actually sent to Discord - not all logs are sent to Discord eventually, depending on the logging configuration.

Everything could be done completely in existing infrastructure basically, as the secure communication from dashboard to bot is already done via the HTTP API in the bot. It would just be another endpoint.

This whole system would also be kept "transport-agnostic" - meaning that the dispatcher and the service.ts file (which is just being used to expose the logging functionality to the bot) could be extracted into a separate repo/package/process if needed.

There is one downside though in comparison to the original idea of using a Cloudflare Worker: If the bot is down, the logging would also be down.

This is a tradeoff I am willing to make, as the dashboard configuration would still work - even if it has no effect because the bot can't use the new config at that moment. However, I have the idea of keeping unsuccessful log events from the dashboard in a queue and retrying them until the bot is back online - but this is a feature for the future. For now, I just want to get the logging system working and then improve it later. Maybe it's not even needed because it's not that important because the dashboard only sends logs for configuration changes and not for core-functionality like ticket creation/closure/deletion, which is the most important part of the logging system. The core functionality is not given when the bot is down, so the logging system not being available in that time has no effect on the core functionality - because it doesn't exist anyways.

The Routing Model

In order to have the most customizability for the config, I wanted to be able to set a different channel for every log action type, but also group related events together and have a default channel for each group. This would allow for a more organized logging system, where similar events are logged in the same channel, while still allowing for specific events to be logged in their own channels if desired.

After all, I decided to make the log config not be in a single document because... MongoDB is not relational and I wanted to have a more flexible and scalable solution, where I wouldn't need to add a new field to the model every time a new log action type is added. Instead, I structured the LogConfig model so I could save one config document per action type. The key is a flattened string of the group and action type, for example ticket.create.__default for the default channel of the ticket.create group - ticket.create.create for the specific channel for the ticket.create action type. This way, I can easily add new action types without having to modify the model.

The thing that I needed to figure out though was what to do with groups where there is only one action type. For example, the ticket.create group only has one action type: ticket.create.create. In this case, it would be redundant to have a group and an action type, as there is only one action type in the group. However, I wanted to keep the structure consistent and not have to special-case this situation - so the default channel for the group would just be unused in such cases. This would also allow for future expansion, where more action types could be added to the group without having to change the structure of the config.

The channel in which the log will be sent after all is determined by first trying to figure out the specific channel ID (and if the log action type is even enabled), then falling back to the default channel for the group.

One thing I gotta still figure out, is how to keep the taxonomy table (groups x action types) in sync between the dashboard and bot. I maybe make a submodule out of the most important things that need to stay in sync. The taxonomy table is basically like a vocabulary of all the log action types and their groups.

The Dispatcher

After I switched from the Cloudflare Workers idea to a more integrated approach, I still had one issue to solve: Queues.

When there are a lot of log events happening in a short amount of time, the bot would be sending a lot of requests to Discord, which could quickly lead to ratelimits being hit. To somewhat mitigate this, I decided to implement an in-process per-channel buffer which groups up to 10 log events per channel and sends them in a single request to the Discord webhook. It uses a sliding window of 5 seconds to group log events together - if there are 3 events within 5 seconds for this channel, they will be sent together in a single message. If there are 10 events within 5 seconds, the queue will be flushed early. This way, I can reduce the number of requests to Discord and avoid hitting ratelimits.

This might seem weird - but if there are many close requests being executed at the same time in a server, they all will generate a log event.

The dispatcher also handles retries for failed requests to Discord - or if the bot went down in the window between a log event creation and the actual sending of the log event. It's single-mechanism: only the sweep re-enqueues stuck events, running every minute. Each event carries an attempts counter, bumped on every transient send failure; once an event hits 10 attempts it's marked dead instead of being retried again. This is to avoid infinite retries in case of a permanent failure (like a webhook being deleted - but more on that later).

Because it can happen that a flush of the queue and a sweep happen at the same time, I added another state to a log event: sending

This is used mainly for the sweep: The sweep will retry events with a state of pending and older than 30 seconds - or sending with a last updated timestamp older than 2 minutes (which indicates a crash mid-flush). Those get re-queued for sending.

The whole delivery state machine looks like this:

pending -> sending -> sent / skipped / dead
sending -> pending (transient error, under attempts cap -> retried by sweep)

Attempts cap (10) only applies to transient send errors — once a batch hits it, event goes dead instead of back to pending. skipped doesn't touch attempts at all (webhook creation failure, thread gone, dead-webhook cooldown not elapsed).

Self-healing Webhooks on Demand

On demand webhooks are something I've seen in the Sapphire bot and I really liked it. Instead of creating a webhook every time a config is saved, the bot will only create a webhook when it is needed - if there isn't a webhook for the channel yet. This also includes threads.

The webhook token is also encrypted at rest, so that it can't be read by anyone who has access to the database. This uses the same system that is also used for user access tokens.

While testing, I came across what I had confidently ignored until now: the edge case error handling for webhooks. What happens if a webhook is deleted? Or if the bot doesn't have permission to create a webhook in the channel? Or if the webhook is deleted and the bot doesn't have permissions to create a new one - and then optionally later gets the permission again?

This might seem like very nitpicky edge cases, but they are very important to handle properly.

There are a few questions that need to be answered here:

1. What happens to the log events that are in the queue for this webhook?
2. What happens to the log events that are created while the webhook is down?
3. What do we try to fix the webhook? How often? For how long? What happens if we can't fix it?
4. How do we let this be known to the server admins/mods?

Until now, the bot would just mark the webhook as dead and stop trying to send log events to it. This is not a good solution, as it would lead to support tickets and confusion. The bot should try to fix the webhook and if it can't, it should let the server admins/mods know that there is an issue with the webhook.

After some thinking with the help of Claude, I came up with a solution that would handle all of these questions.

The DBWebhook model now gets an additional state, recreating, on top of active and dead. The transition from active to recreating is claimed atomically - a findOneAndUpdate that filters on both _id and the expected current state.

Worth noting: a channel's buffer isn't locked while a flush is in flight. The batch itself is safe - flush() swaps buffer.events out synchronously before doing anything async, so one call can never process the same batch twice. But nothing stops a second flush() call for the same channel from starting while the first is still awaiting Discord - the early-flush-at-10 path and the 5s timer path can both fire close together, and the sweep re-enqueuing stale events can kick off another one too. So two flushes for the same channel, each with their own batch, can genuinely hit a terminal Discord error at roughly the same time. That's exactly the scenario the atomic claim guards against: only one of them wins the claim; the other gets null back and backs off, leaving its batch as pending for the sweep to pick up later. Without that guard, both flushes would try to recreate the same webhook and I'd end up with either duplicate webhooks or a race writing stale tokens over fresh ones.

Only terminal Discord errors trigger a recreate attempt in the first place - unknown channel/webhook or missing permissions. Anything else (a 500, a timeout, a random ratelimit hiccup) is treated as transient and just bumps attempts like normal; there's no reason to nuke and rebuild a webhook that's probably fine.

Once a flush claims the recreate, it tries to create a fresh webhook for the channel:

• Success → recreating → active, new webhookId/token saved, and the flush retries the held send once, inline, with the new webhook client. No need to wait for the next batch window - the events that triggered the recreate get delivered immediately if the retry succeeds.
• Failure → recreating → dead, with a deadReason (missing_permissions or unknown_channel) persisted so the dashboard can show why, not just that.

There's also a stale-claim fallback: if a row sits in recreating for more than 30 seconds, the next flush that hits it treats the claim as abandoned (crashed process, most likely) and re-claims it rather than waiting forever on a recreate that's never coming.

For events that show up while a webhook is dead, I didn't want to just silently drop them and I didn't want to spam a full retry either - Discord permissions/channel issues don't usually get fixed instantly. So dead webhooks self-heal on a 15-minute cooldown: the first flush attempt after the cooldown elapses gets to re-claim and try again, same atomic guard as above. Until then, batches for that channel are marked skipped (not dead, not endlessly retried) - attempts isn't touched at all, since the problem isn't the event, it's the webhook.

The one thing I did want to be loud about is the actual recreating → dead transition - that's the point where a human needs to know something's wrong. It fires exactly once per transition (not once per skipped batch after that): a DM to the guild owner, plus a plain non-webhook message dropped into the affected channel (directly by the bot) if the bot can still post there, both carrying the reason. The dashboard also gets webhookState/deadReason surfaced, so a warning dialog can show up there too - it displays a dialog with the things that need attention and stores it in the browser's session storage.

The state machine end to end:

active -> recreating (terminal error, claim succeeds)
recreating -> active (recreate succeeds, held send retried inline)
recreating -> dead (recreate fails)
dead -> recreating (cooldown elapsed [15min], claim succeeds)
recreating -> recreating (claim stale [30s], re-claimed)

Diffing changes without lying about what changed

Once events are actually getting delivered, the next problem is: what do you put in them? "Someone updated the category" isn't useful. "Someone changed the staff role from @Support to @Moderators" is.

My first instinct was the obvious one - diff the whole before/after Mongoose document and see what's different. That lasted about five minutes before I threw it out. Mongoose documents carry updatedAt, internal __v versioning, and whatever else Mongoose likes to touch on save - none of which is a real change a human made. Diffing the whole document means every save produces phantom entries, and phantom entries are worse than no logging at all, because now nobody trusts the log.

So instead: never diff a whole document. Diff an explicit whitelist of fields per model.

The first version of that whitelist was just a flat string[] of field names. That held up for maybe two models before it broke. Some fields carry secrets (a message's password or a guild's encryptedApiKey) - I want the log to say "password was changed" without ever writing the actual password anywhere near Mongo or Discord. Some fields are big nested objects (Form.fields, a whole component tree) - diffing them is fine, but dumping the raw before/after into a Discord embed is both useless (nobody reads a JSON blob in a Discord message) and eventually just breaks the text length limit. And some fields are flat-ish sub-objects (Shield's categories) where I want a per-subkey diff, not "the whole categories object changed."

That's what pushed me toward a small tagged-union type instead of a flat list of strings:

type FieldSpec<T> =
  | keyof T
  | { key: keyof T; opaque: true }
  | { key: keyof T; redacted: true }
  | { key: keyof T; nested: FieldSpec<any>[] };

• A plain key is the default case - whole-value diff, store both sides as-is.
• redacted still runs through the same deep-equal check, so a password rotation is detected and does show up as a change - the value just never survives past the diff itself. It's replaced with "[redacted]" before the change is even recorded, so there's no later "ope, don't render the raw value" step to forget - the raw value simply never reaches Mongo and the embed.
• opaque is diffed and stored in full (the dashboard wants the real structure to show a proper diff view), but flows through a different code path at render time - more on that below.
• nested recurses into a sub-object and diffs only the named sub-keys, dot-joining the path as it goes (categories.ipAddresses: false → true), so nested entries can themselves be redacted or nested again, arbitrarily deep.

The actual comparison is Bun.deepEquals(a, b, true) - strict mode, no new dependency, the same engine that backs bun:test's toEqual. The web dashboard can't use Bun-only APIs, so it ports the identical FieldSpec logic on top of @the-lukez/fast-deep-equal instead - same contract, two engines, so a change diffed on the dashboard side looks exactly like one diffed on the bot side by the time it reaches the database. If there are any conflicts in the future, I will just make a small shared package for the diffing logic.

One more issue worth mentioning: some dashboard endpoints save a whole collection in one request - create, update, delete, and untouched rows, all mixed together in a single payload. Diffing that naively against "what's in the request" isn't enough, because "untouched" rows are still sitting right there in the payload looking like updates. So those endpoints reconcile against the DB first: load what currently exists, diff each incoming row against its existing counterpart by id, and drop anything where nothing actually changed. One log event per entity that genuinely changed, zero events for a no-op save.

Rendering: from a diff to a readable Discord embed

Diffing gets you a list of { key, oldValue, newValue } triples. Turning that into something a server admin actually wants to read is its own problem, and it's got one hard rule underneath it: no raw ID is ever shown bare. A Discord snowflake sitting alone in an embed (728...492) tells nobody anything - it has to become a mention. A Mongo ObjectId is worse, since it's not even resolvable by the reader at all.

Snowflakes are easy - wrap in <@id> / <#id> / <@&id> and Discord does the rest client-side. ObjectIds need an actual DB lookup to become a name (which category, which form, which message), and I didn't want a round trip per changed field. So toEmbed() walks the whole change-set for a doc up front, collects every referenced category/form/message id into a Set, and resolves each model in exactly one batched query - find all ticket categories of which the id matches one of these ids: and so on - before rendering a single line. The embed itself renders synchronously off those resolved maps afterward.

The dashboard even provides a metadata field entry with a targetValue which carries the content to use for a given snowflake or ObjectId on almost every log event.

For everything else there's a small per-field config table instead of scattered ad hoc formatting:

type RenderKind =
  | "plain"
  | "bool"
  | "user"
  | "channel"
  | "role"
  | "userList"
  | "roleList"
  | "channelList"
  | "category"
  | "categoryList"
  | "form"
  | "message"
  | "opaque"
  | "redacted"
  | "countOnly";

Each changed key for a given log group maps to a label + a RenderKind, and the renderer just looks it up - staffRoles is a roleList, claimedBy is a user, a Shield custom-pattern list is countOnly (nobody needs to see the raw regexes in a Discord embed, just "3 → 5"). opaque fields - the ones diffed and stored in full for the dashboard - get suppressed here specifically: the embed shows "Changed — [View in dashboard](...)" instead of trying to cram a component tree into a message. redacted fields render as a fixed "[redacted]" string, same as they were stored.

A handful of fields don't fit a flat before/after line at all, so they get dedicated nested renderers instead of forcing them through the generic table: permission overrides diff as a list of ➕ added / ✏️ changed / ➖ removed lines per role or user, allowed-mentions gets summarized into one line ("parse: everyone | 2 roles, 1 user") instead of dumping the raw object, and authorized bots render as one line per bot with who added it and to which channel. These are maybe 5% of all fields by count, but they're the ones that would look like garbage if forced through the generic oldValue → newValue renderer ("123" → "456").

Fun fact: I basically copied Discord's Permission system with the bitfields to save space in the DB.

Since Ticketon is localized, I must also resolve the locale once per delivery batch, not once per event - the dispatcher already groups everything going out in one flush by guild, so it resolves the guild's locale once and sets it on an ambient AsyncLocalStorage context (withLocale) that every m.*() call inside toEmbed() reads from. Rendering forty events in a burst doesn't mean forty locale lookups.

This reuses the exising li18n setup I built for localizing the bot. Since I added it, manual locale juggling is no longer needed anywhere when doing functional programming - seriously, I'm really proud of it.

What's still open

I'd be lying if I said this was finished. A few things are known trade-offs that I decided to live with for now rather than swept-under-the-rug bugs - and I'd rather name them than pretend the system is perfect.

No batch/request grouping in LogEvent. Right now every changed entity becomes its own LogEvent row. If someone saves a screen on the dashboard that touches 5 messages in one request, that's 5 separate rows in the audit-log list UI. On the Discord side this doesn't really matter - the dispatcher's per-channel batching already collapses them into a single message with up to 10 embeds, so a human reading the channel sees one clean batch. But in the dashboard list, those 5 rows aren't grouped under "one save," and there's no requestId/batchId concept tying them together like there was before. It's not wrong, it's just not as tidy as it could be. Adding a grouping id later is a purely additive change, so I left it out for now.

Single-process assumption in the dispatcher. The in-process buffer, the 5s window, the sweep - all of it assumes exactly one process owns the dispatcher. That's true today: the bot runs as a single process. The moment I add sharding, two processes could both hold a buffer for the same channel and both run the sweep, which reopens all the double-post races the sending lease was built to close. The known answer is to pin the dispatcher to shard 0, or move the sweep behind a Mongo lock so only one process ever runs it. or I just completely extract it and run it separately with the API - communication between it and the shards would then be done via IPC. I didn't build any of these, because building for a shard count I don't have yet is exactly the Cloudflare-Worker mistake from the start of this post, just wearing a different hat.

Retention. No bespoke logging cleanup job. Log events get a 30-day TTL index in Mongo, so old rows expire on their own, and guild-delete purging is folded into the existing deleteData() sweep that already handles GDPR erasure for everything else. One cleanup path for the whole system, not a special case for logs. This was the one of the main points of killing the D1 database - retention stops being a two-system problem the second there's only one database.

Closing

The thing I keep coming back to is how much better the boring answer turned out to be. The first design had a Cloudflare Worker, a D1 database, a Queue, and three signed legs of auth - and it was, on paper, the "proper" distributed-systems way to build this. It lost. Not to a cleverer distributed design, but to ~7 files sitting inside the bot I already had, reusing the signed /dashboard endpoint I'd already built, writing to the database I already had.

It lost the moment I took a step back and assessed the goal again: I needed a single writer and a single schema, not massive scale. The Worker was solving a scaling problem I don't have at the time (a handful of events a minute) while quietly creating a retention problem, an auth problem, and a "how does the edge get the webhook config" problem that I did have. Once the real goal had a name, the architecture more or less picked itself.

So if there's one thing to take from this: before you reach for the impressive architecture, say the actual requirement out loud/write it down and check that it's the one you're building for. Sometimes it is. Mine wasn't.