It’s 2023 and audit logs are a core component of any enterprise product offering. As simple as they seem, audit logs can be tricky to implement.
Having made this feature twice as part of my work at Infisical, I discuss everything about audit logging in this article and specifically how to ship them properly for yourself.
What are audit logs?
To begin, audit logs are a centralized stream of user activity used by security and compliance teams at enterprises to monitor information access in the event of any suspicious activity or incident review. At first glance, they are entries consisting of events, time-stamps, and payloads. On a deeper level, however, they can be seen as summaries that, when taken in tandem, capture a frame-by-frame narrative of what happened at any given period of time in great detail (we call this a fine-grain audit trail).
Making great audit logs
In this section, we outline the data structure we’d expect from an audit log and the principles we wish to uphold.
The general data structure
As a reminder, audit logs are purpose-built for security and compliance teams to monitor and inspect the context of an incident at a given point in time. Given this, a good set of audit logs should, at the every least, contain fields that answer your basic interrogative pronouns:
Event: What happened? — For Infisical, this was, for example, an action associated with a secret such as it being fetched or created.
Actor: Who triggered the event? This may be the name, email, or identifier for the entity responsible for performing or causing the underlying event.
Timestamp: When did the event occur? — Source (User agent + IP): Where did the event occur? If you’re building an API, both user information and IP address can typically be obtained from the request itself. Frameworks like Express, for instance, expose the request’s user agent under req.headers[“user-agent”] and req.ip .
Metadata: Additional data to provide context for each event. In our case, this could’ve been the path at which a secret was fetched from etc.
Beyond these fields, it can be useful to store the following data (these may or may not be applicable):
Source Type: The source category of the event. In our case, requests could be made from a Web UI, CLI, Kubernetes Operator, or Other (i.e. anywhere else). We’ll discuss this more later but including such a field helps from a querying standpoint.
Description: A human-readable description of the action taken. The defining characteristics.
Status: Like HTTP requests, it’s possible for events to succeed or fail. If applicable and you have a way to detect this status information, you should include it in the entry. This would be helpful to identify, for instance, if someone attempted to access a forbidden resource.
A good audit logging system should not only record event data but also consider various usability criteria for the teams inspecting the logs (i.e the end users) and engineers of the system itself:
Immutable: Users of the system should only have the ability to read data from it and not write to it. Having write-ability would compromise the integrity of the audit logging system since it must reflect the exact state of events in the past.
Queryable: Users of the system must be able to efficiently search through the audit logs by applying any one or more filters; filtering should be quick. In case of a security incident, an administrator should be able to filter out all events where actor A performed action B from source C between the dates D and E.
Exportable: Data must be exportable and/or streamable to a dedicated logging solution like Splunk, AWS, etc. This is useful for security teams to centralize their security logs to be analyzed externally.
Lastly, you may consider some additional nice-to-have qualities:
Lightweight: Audit logs should capture brief summaries, snippets, and/or references to data rather than include all of it. So, if you find yourself copying an entire data structure over as if you’re capturing a database backup/snapshot, then you’re probably doing it wrong (e.g. you don’t need to attach every detail about the actor for an event; you may only need their identifier to link them to the event).
Parsable: A great audit log data structure should also be easy to parse and, more generally, work with. The shape and structure of your data can have implications on how easy it is to render it over in the platform UI as well as to manipulate for some future engineering endeavor.
How we did it at Infisical
When we first built the audit logging functionality for Infisical, we didn’t fully consider the target user at hand.
Instead, we viewed the objective in an overly-simplified way: create a chronologically-ordered ledger of events for teams to inspect the past. We also mistook audit logs for complete snapshots of data and state that, in retrospect, should be the responsibility of database backups/snapshots.
As a result, we ended up with a impractical audit logging system that took us back to the drawing board.
The data structure we used
After giving it a lot of thought, we opted for the following audit log schema:
export interface IAuditLog {
actor: Actor;
organization: Types.ObjectId;
workspace: Types.ObjectId;
ipAddress: string;
event: Event;
userAgent: string;
userAgentType: UserAgentType;
expiresAt: Date;
}
const auditLogSchema = new Schema<IAuditLog>(
{
actor: {
type: {
type: String,
enum: ActorType,
required: true
},
metadata: {
type: Schema.Types.Mixed
}
},
organization: {
type: Schema.Types.ObjectId,
required: false
},
workspace: {
type: Schema.Types.ObjectId,
required: false
},
ipAddress: {
type: String,
required: true
},
event: {
type: {
type: String,
enum: EventType,
required: true
},
metadata: {
type: Schema.Types.Mixed
}
},
userAgent: {
type: String,
required: true
},
userAgentType: {
type: String,
enum: UserAgentType,
required: true
},
expiresAt: {
type: Date,
expires: 0
}
},
{
timestamps: true
}
);
Fundamentally, we split the necessary components into 3 parts adjusted to our needs:
Actor: Since events in Infisical could be triggered by human and non-human (i.e. services) actors, we added actor type to the schema. Moreover, since human and non-human actors had different identifier metadata like email, user ID, service ID, etc., we also created a well-typed, mixed type metadata field to store this information for each actor. The defining thought process here was query-ability.
Event: The name of the event and any associated metadata unique to each event.
Other: Request information such as the IP address and user agent of the inbound request as well as a timestamp of when it came in.
Equipped with this data structure, creating and querying for audit logs became easy as cake.
The UI and UX
As you’d expect, we organize audit logs into a paginated table view, equipped with filters that users can apply together to narrow their search.
We made a ton of improvements since our first iteration of audit logs not limited to:
Filter options: You can filter audit logs by event, user, source, date, or any combination of these filters.
Metadata: We display only relevant metadata and nothing excess. For instance, we don’t store and perform any crypto operations on audit logs when it comes to secrets; we just offload that to another part of the application that’s responsible for it.
Timestamps: We format time-stamps in “YYYY-MM-DD hh:mm am/pm” format.
Closing thoughts
Audit logs are a high-demand feature when it comes to selling enterprise software and it’s important to get them right. By choosing a versatile data structure and considering how they are used in practice, you can design great audit logs for your software that help users gain insight for security reviews and incidents.
— -
Infisical — The open source secret management platform
Infisical (8.4K+ ⭐️)helps thousands of teams and organizations store and sync secrets across their team and infrastructure.
GitHub repo: https://github.com/Infisical/infisical
Top comments (0)