DEV Community: Xabi Errotabehere

When Should We Expand into a New AWS Region?

Xabi Errotabehere — Tue, 26 Mar 2024 00:02:35 +0000

As the CTO of Hivelight, I've been grappling with the pivotal decision of when to expand into new AWS regions. This isn't just a technical matter; it's about ensuring our company's growth, resilience, and ability to serve our clients effectively. Drawing from my experiences and insights from the vibrant community, let's explore the nuances of this decision.

Why expand?

In today's interconnected world, where our users are spread across various regions (ANZ, North America, the Caribbean, and Europe), expanding into new AWS regions is no longer a luxury but a strategic necessity. By doing so, we unlock several benefits that directly impact our startup's performance and competitiveness:

Reduced Latency for Enhanced User Experience
Imagine a lawyer using our platform to access critical matters or collaborate with colleagues. Reduced latency means quicker access and seamless interactions, leading to higher user satisfaction and retention.
Improved Fault Tolerance and High Availability
As a productivity platform, downtime is simply not an option. By duplicating our infrastructure across multiple AWS regions, we increase our fault tolerance and ensure that our services remain accessible even in the face of regional outages or disruptions.
Compliance and Data Residency Considerations
In the legal sector, where data privacy and regulatory compliance are paramount, operating in multiple AWS regions allows us to adhere to data residency requirements and navigate complex regulatory landscapes.

Signs Pointing Towards Expansion

Expanding into new AWS regions shouldn't be a hasty decision but a well-thought-out strategy informed by data and market insights. Here are some key indicators that suggest it's time for us to broaden our AWS footprint:

User Base Distribution Analysis
By closely monitoring the distribution of our user base, we can identify regions where there's a concentration of users. If a significant portion of our users is clustered in a specific geographic area, it's a strong signal that we need to establish a local presence to optimize performance and responsiveness. As of today, most of our users are in the ANZ region, but our expansion plans are in North America.
Latency Metrics and Performance Monitoring
Tracking latency metrics and performance indicators across different regions provides us with valuable insights into user experience. Consistently high latency for users in certain regions indicates a need for local infrastructure to mitigate delays and improve service quality. For instance, the roundtrip from New Zealand to us-east-1 (AWS) for a lambda function is 500ms, but can vary from 200ms to 3 seconds. No user has expressed that the system is slow, better yet, our users are happy with the responsiveness.
Historical Downtime and Outage Analysis
Analyzing past downtime incidents and regional outages helps us identify patterns and vulnerabilities. If we notice recurring issues or a particular region's susceptibility to disruptions, it would signal the importance of diversifying our AWS footprint to enhance reliability and resilience, so at least only a fraction of our customers would be impacted by such an event. So far, we haven't seen any outage that impacted our production environment, fingers crossed.
Strategic Market Expansion Plans
Our startup's growth trajectory and market expansion plans also influence our decision to expand into new AWS regions. If we're targeting new geographical markets or experiencing rapid growth in specific regions, establishing a local presence ensures that our infrastructure can scale seamlessly to meet growing demand. That is why we are using in us-east-1 (AWS) because this is where we see the biggest growth potential.

To sum up, expanding into new AWS regions is a strategic move that requires careful consideration, data-driven insights, and collaboration. By assessing user distribution, latency metrics, downtime incidents, and market expansion plans, we can pinpoint opportune moments to diversify our AWS footprint and unlock new avenues for growth and innovation. We're not there yet however, we need to be ready for that.

This isn't just about technical infrastructure; it's about empowering our company to thrive in an ever-changing landscape. We can chart a course towards success and ensure that our platform continues to provide the platform that every law practice needs to grow, do its best work, and help more people.

How we manage user and system events at Hivelight

Xabi Errotabehere — Thu, 30 Mar 2023 20:11:21 +0000

Think of Hivelight as Jira for law firms.

As a productivity management system, Hivelight deals with lots of user events generated by users creating tasks, moving tasks across the Kanban board, assigning them and a whole lot more actions. All these events need to be stored as need to be able to provide a history of what happened and when it happened for every task, comment, milestone etc, a user has touched or seen.

We also need to be able to react to these events. For instance, when a user assigns a task to another user, we need to let the assignee now this task has been assigned to them. Or when a task is completed, we need to update the progress status of the milestone it belongs to.

Fortunately, our application is serverless and we are using the Serverless Framework to orchestrate the deployment. This makes building event-driven applications very easy.
Most of our data is stored in DynamoDB tables and DynamoDb Streams will tremendously help us. We will also use EventBridge to create an application event bus.

I will try to explain how we deal with events by over simplifying out setup.

Producing events

Let's imagine a user has been working on a task and marks it as done. This updates the task's status on DynamoDB's Task table. This in turn, puts the old task with the old status (OLD_IMAGE) as well as the new task with the new status (NEW_IMAGE), into the DynamoDB Stream.
This means it is easy for us to determine what has changed by creating a Lambda function that ingests that DynamoDB Stream.

Let's imagine this DynamoDB Stream record:

{
    OLD_IMAGE: {
        id: 1234,
        name: "My task",
        status: "IN_PROGRESS",
        lastModifiedBy: "Alice",
        lastModifiedDate: "2023-02-21",
    },
    NEW_IMAGE: {
        id: 1234,
        name: "My task",
        status: "DONE",
        lastModifiedBy: "Bob",
        lastModifiedDate: "2023-03-29",
    }

}

We can create a simple algorithm that will make sense of what has changed.

Here, we can determine that the status has changed from IN_PROGRESS to DONE. We also note that the control fields have been updated. This means we can determine not only that the status has been updated, but who has updated it, and when.

This information is enough for us to create an event. That is what the Lambda function does by ingesting changes to the DynamoDB table and creating respective events.

Using the DynamoDB Stream record example above, we could create an event that looks like that:

{
    name: "task.completed",
    taskId: "1234",
    user: "Bob",
    date: "2023-03-29"
}

Storing events

Before sending this event to the event bus, let's save it so we can query it later.

This action is simply done by saving this event into a another DynamoDB table called Event. This table has many indices to allow us to very quickly query the events by user, date, event name, etc.

As a side note, I have been interested in Amazon Timestream since its private launch was first announced. It has finally been generally available but its costs and write operations latency made it unsuitable for our budget and technology. We're are being charged by the millisecond when using Lambda functions and a 500ms latency for every write request to Amazon Timestream doesn't cut it for our stage of application growth.
In comparison, storing and querying the event in DynamoDB table is fast, simple and cost effective for our workload type.

Ingesting events

Once the event is stored into the DynamoDB Event table, we can then have a Lambda function that ingests the DynamoDB Stream record as an event, and publishes it to the EventBridge's event bus we named ApplicationEvents.

We can now listen to this event anywhere in our application. For instance, we want to trigger a Lambda function when a task status gets updated, we can specify that in our Serverless Framework definition file (serverless.yml) as such:

  onTaskCompleted:
    handler: src/functions/onTaskCompleted.handler
    events:
      - eventBridge:
          eventBus: arn:aws:events:${aws:region}:${aws:accountId}:event-bus/ApplicationEvents
          pattern:
            source:
              - task.completed

Whenever a task's status is completed, the onTaskCompleted Lambda function will be executed with an event that looks like:

{
  version: "0",
  id: "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
  "detail-type": "ApplicationEvent",
  source: "task.completed",
  account: "123412341234",
  time: "2023-03-29T17:31:09Z",
  region: "us-east-1",
  resources: [],
  detail: {
    name: "task.completed",
    date: "2023-03-29"
    taskId: 1234,
    user: "Bob"
  }
};

Neat!

We have seen how creating, publishing and subscribing to application events is made easy with DynamoDB Stream, Lambda and EventBridge event bus. This opens so much possibility by decoupling event producers from event consumers. This allows for cross-service, cross-application and even cross-AWS-account reliable, cheap and efficient communication.

But what about querying?

Querying events

We have all the events sitting on a table, and that can help us extract valuable information about user behaviour, application usage, etc.

Let's imagine a scenario where we would like to know how much task activity is produced by a user as well as how many tasks have been changing status per day for a given user.

We could easily get that by querying out Event table with these queries:

Get all activity for a user where from and to are timestamps:

{
    TableName: "Event",
    KeyConditionExpression: "user = :user AND #date BETWEEN :from AND :to",
    ExpressionAttributeValues: {
      ":user": user, 
      ":from": from,
      ":to": to
    },
    ExpressionAttributeNames: {
      "#date": "date",
      "#name": "name"
    },
    ProjectionExpression: "#date"
  };

Get all task-related activity for a user where from and to are timestamps:

{
    TableName: "Event",
    KeyConditionExpression: "user = :user AND #date BETWEEN :from AND :to",
    ExpressionAttributeValues: {
      ":user": user, 
      ":from": from,
      ":to": to,
      ":event1": "task."
    },
    ExpressionAttributeNames: {
      "#time": "time",
      "#date": "date"
    },
    ProjectionExpression: "#date",
    FilterExpression: "begins_with (#name, :event1)"
  };

Get how many tasks have been completed or started for a user where from and to are timestamps:

{
    TableName: "Event",
    KeyConditionExpression: "user = :user AND #date BETWEEN :from AND :to",
    ExpressionAttributeValues: {
      ":user": user, 
      ":from": from,
      ":to": to,
      ":event1": "task.completed",
      ":event2": "task.started"
    },
    ExpressionAttributeNames: {
      "#date": "date",
      "#name": "name"
    },
    ProjectionExpression: "#date",
    FilterExpression: "begins_with (#name, :event1) OR begins_with (#name, :event2)"
  };

We are only retrieving the time of the event via the ProjectionExpression value as we are not interested in getting the actual event. We are only counting events per day by doing something like:

const dailyCounts = events.reduce((acc, event) => {
    const date = event.date;
    if (!acc[date]) {
      acc[date] = 1;
    } else {
      acc[date]++;
    }
    return acc;
  }, {});

This should give us something like

{
...
    "2023-03-27": 23,
    "2023-03-28": 45,
    "2023-03-29": 22,
...
}

This makes it easy to create activity visualization such as the GitHub activity-like calendar chart to better understand activity patterns and trends of specific user, or entity buy querying the Event table and counting the number of occurrences per day.

For instance, the screenshot below shows all activity events daily produced by me:

Or show my log-in events only

What do you think? What are you using to record user and system generated events. Have you been successful in implementing Amazon Timestream? Let me hear your thoughts.

I've created a full-stack user permission model, should I go open-source?

Xabi Errotabehere — Thu, 02 Feb 2023 23:40:20 +0000

I have been working on the Hivelight application for several months, managing user roles and permissions. Initially, I used a simple approach to handle roles such as Owner, Admin, User, and Guest, each triggering a specific piece of code. This method worked well until I had to deal with more complex permissions. That's when I searched for an elegant solution to handle roles and permissions both in the back-end and front-end.

The solution needed to take into account organizations that have multiple workspaces, which in turn have multiple matters and multiple tasks. There are other relationships to take into account, but I'll keep them out of this article for brevity.

The goal is to design a fine-tuned permission model that not only considers the user's role and the resource being accessed but also the context and provides access if certain conditions are met. The solution should consider three main concepts:

The solution has to take into account 3 main concepts:

A resource
One or more actions to be performed against it
Whether these actions are permitted or not

The solution

I was inspired by the AWS IAM policies and came up with the HRL (Hivelight Resource Locator) concept to describe a resource or its path in the resource hierarchy.

For example, a task located in organizationId/workspaceId/matterId/taskId would translate to hrl:organizationId:workspaceId:matter:matterId:task:taskId.

To describe all tasks within a workspace, the HRL would be hrl:organizationId:workspaceId:matter:*:task:*, and to describe all matters in a workspace, it would be hrl:organizationId:workspaceId:matter:*.

Next, the solution needs to define the actions a policy holder can perform against the identified resource. These actions should consist of an action verb (create, read, update, delete) and a resource name (Task, TaskAssignee, Matter, MatterTag, etc.).

By combining these three concepts, the solution would look like a policy document like this:

    - resource: hrl:[organizationId]:[workspaceId]:matter:*
      actions:
        - updateMatterStatusMessage
        - createMatterTag
        - createMatterNote
      effect: allow

This allows a policy holder to update a matter status message, create a matter tag, and create a matter note on any matter in a given workspace.

Multiple statements can be combined to create a rich and fine-tuned permission model, like this:

    - resource: hrl:[organizationId]:[workspaceId]:matter:*
      actions: “*”
      effect: deny
    - resource: hrl:[organizationId]:[workspaceId]:matter:[matterId]
      actions:
        - readMatter
      effect: allow

This allows a policy holder to read a specific matterId in a workspace but deny access to any other matter

The library

The solution includes a full-stack library to handle actions and determine if a specific action can be taken against a specific resource. The code for this would look like this:

permission.can("actionResource")

This returns true or false depending on the permission.

For example:

<div>
  {permission.can("updateMatterStatusMessage") ? (
    <StatusUpdate matterId={matter.id} />
  ) : undefined}
</div>

<div>
  {permission.cannot("createMatter") ? (
    <p>You don't have enough privilege to create a matter</p>
  ) : undefined}
</div>

or like that in the backend

if (permission.cannot("createMatter")) {
  throw new Error("Forbidden")
}

To provide context for the logic that makes the "can/cannot" decision, we need to construct an HRL that describes the resource being dealt with, the policy or policies held by the current user, and the context of who the current user is and what other entities are involved.

Consider this:

// constructing the hrl
const hrl = [
  "hrl", 
  currentOrganizationId, // id is 123
  currentWorkspaceId // id is ABC
].join(":");

// constructing the context object
const context = {
   currentUser
}

// creating the permission object assuming
// the user object contains the policy document
const permission = new Permission(hrl, currentUser.policy, context);

// ... then somewhere in the code
if (permission.cannot("updateMatterStatusMessage")) {
  throw new Error("Forbidden")
}

In the example, the HRL is constructed as hrl:123:ABC, where 123 and ABC are the IDs of the current organization and workspace, respectively. The context object contains information about the current user, and this information, along with the user's policy, is passed to create a permission object.

Imagine the policy has the following statements:

- name: WorkspaceMember
  statements:
    - resource: hrl:123:ABC
      actions: "*"
      effect: allow
    - resource: hrl:123:ABC:matter:*
      actions: "*"
      effect: allow

The code above will not throw an error because this user can perform any action actions: "*" on every matter resource: hrl:123:ABC:matter:*

Now, imagine the policy has the following statements:

- name: WorkspaceMember
  statements:
    - resource: hrl:123:ABC
      actions: "*"
      effect: allow
    - resource: hrl:123:ABC:matter:*
      actions: "*"
      effect: allow
    - resource: hrl:123:ABC:matter:*
      actions: 
        - updateMatterStatusMessage
        - deleteMatter
      effect: deny

The code above would throw an error because the user is denied updateMatterStatusMessage action on every matter

The library mentioned here is lightweight and has been used successfully in production for both the frontend and backend.
It comes with more features not developed here, such as conditional permissions.

It has the advantage of keeping the front and backend logic unchanged when the permission model changes and allowing roles and permission models to be applied on a per-user basis, with each user holding a policy that suits them.

Another big advantage is also that the logic and the policy are kept separate. This enables the usage of different libraries (i.e. Java back-end and JS front-end) to use the same policy documents.

I am considering making the library open-source. If you have used similar libraries before, what are their shortcomings, and do you think the model described here can solve them? Would you be interested in taking a look at the library? What role and permission library do you use?