DEV Community: Rob Sutter

Applying test-driven development to your database

Rob Sutter — Thu, 03 Feb 2022 16:31:33 +0000

Test-driven development (TDD) is a popular software development methodology that aims to reduce defects and improve the ease of maintaining code. In this post, you learn how to apply TDD to write queries and expressions with the Fauna Query Language (FQL). You learn a general pattern for TDD with FQL, then apply that pattern to implement a Split() function that accepts string and delimiter parameters and returns an array of substrings “split” at each occurrence of the delimiter.

Pre-requisites

To follow along with this post you must have access to a Fauna account. You can register for a free Fauna account and benefit from Fauna’s free tier while you learn and build, and you won’t need to provide payment information until you are ready to upgrade.

This post assumes you are familiar enough with Fauna to create a database and client key. If not, please complete the client application quick start and return to this post.

This post also assumes you have Node.js v12 or later in your development environment. This post uses Jest and JavaScript, but you can apply the principles to your preferred test tooling in any supported language.

The source code used in this post is available in Fauna Labs.

TDD overview

TDD begins with writing a single failing test. For example, if you are implementing a Split() function that accepts a string and a delimiter and returns an array of strings “split” at each occurrence of the delimiter, you would test that Split("127.0.0.1", ".") is equal to ["127", "0", "0", "1"]. When you first run your test suite, the test fails, because you have not yet implemented the Split() function.

Next, you implement the functionality of a single “unit” of code, in this case, the Split() function. You run your tests as you make changes, and when the tests pass, you have implemented the code to cover the specific use case that the test specifies.

With passing tests, you can add more tests to cover additional use cases, or you can refactor your existing code for performance or readability improvements. Because your tests are in place and passing, you can make changes with more confidence that your code is correct.

Setting up your environment

Setting up your database

Create a new database in your Fauna account and create a key for that database with the Server role. Save the key to add to your application.

Configuring Jest

Create a new Node.js application in an empty directory and install the Fauna JavaScript driver.
```
npm init --yes
npm install faunadb
```

Add Jest to your application.

npm install --save-dev jest babel-jest @babel/core @babel/preset-env

Edit package.json to add the following “test” and “test:watch” scripts.

{
  "scripts": {
    "test": "jest",
    "test:watch": "jest --watchAll"
  }
}

Create a file babel.config.js in the root directory of your project with the following content.

module.exports = {
  presets: [['@babel/preset-env', {targets: {node: 'current'}}]],
};

Create a file jest.config.mjs in the root directory of your project with the following content.

export default {
  collectCoverage: true,
  coverageDirectory: "coverage",
  coverageProvider: "v8",
  setupFiles: ["<rootDir>/.jest/setEnvVars.js"],
};

Create a file .jest/setEnvVars.js and paste the following code, replacing with the value of your admin key and db.fauna.com with your Region Group endpoint. Add this file to your .gitignore to avoid committing secrets to your repository.
```
process.env.FAUNADB_ADMIN_KEY = '<secret_key>';
process.env.FAUNADB_DOMAIN = 'db.fauna.com';
```
Run npm run test -- --passWithNoTests to check that you have configured your application to use Jest. You should receive a response “No tests found, exiting with code 0.”

Testing partial FQL expressions

You test partial FQL expressions from JavaScript by writing JavaScript functions that generate FQL expressions. You then export these functions so that both your code and your tests can import and use them. This supports the “don’t repeat yourself” or “DRY” principle of software development in your FQL queries.

General pattern

The following five steps form the general pattern for testing partial FQL expressions:

Define the input to your expression as one or more constants.
Define the expected output of your expression as a constant.
Construct your FQL expression.
Evaluate your FQL expression in Fauna and save the actual output.
Compare the actual output to the expected output.

Your test passes when the comparison in the final step is successful. This comparison may be equality, inequality, or any other comparison allowed by your testing framework. In Jest, these comparisons are called matchers; your testing framework may have different terminology.

Preparing your test harness

Make a stub for your implementation by creating a file named fauna/lib/split.js and pasting the following code.

export function Split(str, sep) {
  return undefined;
};

This gives your test an undefined implementation to call, guaranteeing your test fails.

Next, create a file fauna/test/split.test.js in your project. Since you run your tests against an actual Fauna environment, you must import the Fauna JavaScript driver and create a new Fauna client by adding the following code.

import faunadb from 'faunadb';

const faunaClient = new faunadb.Client({
  secret: process.env.FAUNADB_ADMIN_KEY,
  domain: process.env.FAUNADB_DOMAIN || "db.fauna.com",
  port: parseInt(process.env.FAUNADB_PORT) || 443,
  scheme: process.env.FAUNADB_SCHEME || "https",
  checkNewVersion: false,
});

Next, import the Split() function stub that you want to test.

import { Split } from '../lib/split';

Copy and paste the following general framework code into split.test.js.

test('<description>', async () => {
  // 1. Define the input to your expression as one or more constants.
  const input = ...;

  // 2. Define the expected output of your expression as a constant.
  const expected = ...;

  // 3. Construct your FQL expression.
  const expr = ...;

  // 4. Evaluate your FQL expression in Fauna and save the actual output.
  const actual = await faunaClient.query(expr);

  // 5. Compare the actual output to the expected output.
  expect(actual).toEqual(expected);
});

At this point, you have a complete test harness, but your tests will not yet run because of the placeholder values. In the next section, you replace these with real values based on the specified behavior of your code.

Create a single failing test

The following examples specify the expected behavior of your Split() implementation.

Split(ip_address, '.') correctly separates an IPv4 address into four octets.
When delimiter is not found, Split(string, delimiter) returns an array with one element, string.
When the string contains only delimiter, Split(string, delimiter) returns an empty array.

Test	string	delimiter	Expected result
1	`'127.0.0.1'`	`'.'`	`['127', '0', '0', '1']`
2	`'Hello, world!'`	`'.'`	`['Hello, world!']`
3	`'................'`	`'.'`	`[]`

To implement the first test, modify the general framework code in fauna/test/split.test.js with the properties from your first specification. Your implementation may vary but should be similar to the following code.

test('Split correctly separates an IPv4 address into four octets', async () => {
  // 1. Define the input to your expression as one or more constants.
  const input = {
    string: '127.0.0.1',
    delimiter: '.',
  };

  // 2. Define the expected output of your expression as a constant.
  const expected = ['127', '0', '0', '1'];

  // 3. Construct your FQL expression.
  const expr = Split(input.string, input.delimiter);

  // 4. Evaluate your FQL expression in Fauna and save the actual output.
  const actual = await faunaClient.query(expr);

  // 5. Compare the actual output to the expected output.
  expect(actual).toEqual(expected);
});

The previous code expects Split() to split the string ‘127.0.0.1' into an array of four strings at each decimal point/period, yielding ['127', ‘0', '0', '1'].

Before moving on to implement Split(), check that your test runs using npm run test. It’s okay if it fails. This is the expected behavior since you haven’t implemented the functionality of Split()!

If your tests do not yet run, check that you have included all parts of the test harness and the test itself. Your complete fauna/test/split.test.js should look as follows.

import faunadb from 'faunadb';

import { Split } from '../lib/split';

const faunaClient = new faunadb.Client({
  secret: process.env.FAUNADB_ADMIN_KEY,
  domain: process.env.FAUNADB_DOMAIN || "db.fauna.com",
  port: parseInt(process.env.FAUNADB_PORT) || 443,
  scheme: process.env.FAUNADB_SCHEME || "https",
  checkNewVersion: false,
});

test('Split correctly separates an IPv4 address into four octets', async () => {
  // 1. Define the input to your expression as one or more constants.
  const input = {
    string: '127.0.0.1',
    delimiter: '.',
  };

  // 2. Define the expected output of your expression as a constant.
  const expected = ['127', '0', '0', '1'];

  // 3. Construct your FQL expression.
  const expr = Split(input.string, input.delimiter);

  // 4. Evaluate your FQL expression in Fauna and save the actual output.
  const actual = await faunaClient.query(expr);

  // 5. Compare the actual output to the expected output.
  expect(actual).toEqual(expected);
});

Implementation

Now that your test harness is set up, run your tests in watch mode.

npm run test:watch

In watch mode, test tooling watches your source and test files and runs the relevant test suite when changes are detected. You immediately see the results of running the test suite whenever you save your work, which tightens the developer feedback loop and improves your productivity.

With your tests running in watch mode, it’s time to implement the actual Split() function so that your tests pass. Replace the stub you created earlier in fauna/lib/split.js with the following code. This implementation uses the built-in FQL action FindStrRegex to split the parameter str at each occurrence of sep.

import faunadb from 'faunadb';

const q = faunadb.query;
const {
  Concat,
  FindStrRegex,
  Lambda,
  Map,
  Select,
  Var
} = q;

export function Split(str, sep) {
  return Map(
    FindStrRegex(str, Concat(["[^\\", sep, "]+"])),
    Lambda("res", Select(["data"], Var("res")))
  );
};

When you save your file, Jest automatically re-runs the test suite, and this time, it passes. Congratulations, you’ve written your first TDD with FQL!

Review and next steps

In this post, you learned how to apply TDD to write and test FQL expressions. You wrote failing tests for a Split() function that accepts string and delimiter parameters and returns an array of substrings “split” at each occurrence of the delimiter. Finally, you implemented the Split() function while running unit tests in watch mode.

To test your knowledge, try implementing tests for the remaining two specifications. Once you have those tests in place and passing, try to refactor the implementation of Split(). Notice how TDD allows you to safely refactor your code without introducing regressions.

For more tooling and sample code, check out Fauna Labs on GitHub. We can’t wait to see what you test and build!

Migrating your GraphQL schema with Fauna

Rob Sutter — Fri, 10 Sep 2021 19:00:26 +0000

Modern applications frequently change as you deliver features and fixes to customers. In this series, you learn how to implement migrations in Fauna, the data API for modern applications.

The first post in this series introduces a high-level strategy for planning and running migrations. The second post demonstrates how to implement migration patterns using user-defined functions (UDFs) written in the Fauna Query Language (FQL). In this post, you learn specific considerations for migrating Fauna databases that you access via GraphQL and how to apply the techniques from the previous posts.

All of the code in this series is available on GitHub in Fauna Labs.

Migration scenario

This post implements the same migration scenario you used in the previous post. Your Fauna database has a collection for firewall rules and you have an updated requirement to manage inbound traffic from an arbitrary number of IP address ranges. To satisfy this requirement, you must migrate the ipRange field type from an FQL string to an FQL array of strings.

Pre-requisites

To follow along with this post you must have access to a Fauna account. You can register for a free Fauna account and benefit from Fauna’s free tier while you learn and build. You do not need to provide payment information until you upgrade your plan.

You do not need to install any additional software or tools. All examples in this post can be run in the web shell and the GraphQL playground in the Fauna dashboard.

This post assumes you have read and understood the previous posts in this series.

Create and populate your database

Create another new database in the Fauna dashboard. Do not select the "Pre-populate with demo data" checkbox, and do not re-use the database from the previous post.

Save initial-schema.graphql to your computer. In your new database in the Fauna dashboard, select the GraphQL tab, choose Import Schema, and upload initial-schema.graphql.

type FirewallRule {
  action: String!
  port: Int!
  ipRange: String!
  description: String
}

Navigate to the Collections tab and notice that Fauna creates an empty FirewallRule collection.

Navigate to the Functions tab and choose New Function. Enter migrate_firewall_rule as the Function Name and leave the Role set to the default None. Paste the following FQL in the field Function Body and choose Save to create your UDF. Note that this is the same UDF you created in the previous post.

migrate_firewall_rule

Query(
  Lambda(
    ["firewall_rule_ref"],
    Let(
      { 
        doc: Get(Var("firewall_rule_ref")),
        ipRange: Select(["data", "ipRange"], Var("doc"))
      },
      If(
        IsArray(Var("ipRange")),
        Var("doc"),
        Update(
          Var("firewall_rule_ref"),
          { data: { ipRange: [Var("ipRange")] } }
        )
      )
    )
  )
)

Return to the GraphQL tab, copy the mutation from populate-firewall-rules.graphql, and paste it into the query editor. Choose the "play" button and Fauna runs your mutation and returns the _id and description for each sample firewall rule.

Return to the Collections tab and confirm that your FirewallRule collection now contains three documents.

Encapsulation

Encapsulation is built into GraphQL via strongly typed queries and mutations. Choose Schema in the GraphQL Playground and review the modified schema that Fauna generates for your database. Note that Fauna makes the following relevant modifications to your schema:

Copies the type FirewallRule you provide to an input FirewallRule.
Adds _id and _ts fields to the type FirewallRule you provide.
Generates one GraphQL query - findFirewallRuleById.
Generates three GraphQL mutations - createFirewallRule, updateFirewallRule, and deleteFirewallRule.

You can override or redefine everything that Fauna generates for you, including GraphQL input types. You can also specify UDFs as custom resolvers for both queries and mutations. Together, these characteristics form the basis of your migration strategy with GraphQL.

Migrating in steps

You must create UDFs for each relevant query and mutation the first time you perform a migration with GraphQL. Except for the final step, each individual step leaves your database in an equivalent state that does not require any downtime.

For your first migration, you perform the following steps in order:

Specify a resolver for your query.
Specify your input.
Specify a resolver for each mutation.
Migrate in one step:
- Modify your UDFs to accept the new shape of your data.
- Modify your type and input definitions in your GraphQL schema and replace your schema in Fauna.

You perform subsequent migrations by modifying the relevant UDFs and uploading a new schema in a single step.

Tip: Use tooling for the final step! Fauna provides the Fauna Schema Migrate tool and a Serverless Framework plugin to help you manage your resources in Fauna as code.

For all migrations, it is less risky if you perform your migration during an application downtime period. If you are using infrastructure as code (IaC) tools, a single migration should typically take only a few seconds.

Step one - Specify your first resolver

The first change you make to your database is to explicitly specify a UDF as a resolver for your query. You modify your query before your mutations because it does not require specifying an input. This way you change only one thing at a time and check for correctness, supporting the principle of migrating in steps.

Return to the Functions tab and again choose New Function. This time enter find_firewall_rule_by_id as the Function Name and leave the Role set to the default None. Paste the following FQL in the field Function Body and choose Save to create your UDF.

find_firewall_rule_by_id

Query(
  Lambda(
    ["id"],
    Get(
      Ref(Collection("FirewallRule"), Var("id"))
    )
  )
)

Open your schema and explicitly define the findFirewallRuleByID query by adding the following definition:

type Query {
  findFirewallRuleByID(id: ID!): FirewallRule @resolver(name: "find_firewall_rule_by_id")
}

There are two points to note about this definition:

Everything except for @resolver(name: "find_firewall_rule_by_id") is copied directly from the schema that Fauna generates. This means you are not creating a new query, but are directing an existing query to use a specific UDF with the @resolver directive.
The @resolver directive takes one parameter, name, whose value is the name of the UDF Fauna invokes to process the query.

Save your schema and return to the GraphQL tab. Choose Replace Schema, choose Replace, and select your modified schema.

Step two - Create an input

When you upload a GraphQL schema, Fauna copies any type definitions to a corresponding input definition. If you explicitly define this input in your schema, Fauna uses the definition you specify.

Copy the following input FirewallRuleInput definition and add it to your schema:

input FirewallRuleInput {
  action: String!
  port: Int!
  ipRange: String!
  description: String
}

Note this definition is identical to the generated schema in your Fauna dashboard. For your own migrations, you can copy the input definition from the schema Fauna displays in the GraphQL tab.

Save your schema again, return to the GraphQL tab, and replace your schema with the newest version.

Step three - Specify resolvers for mutations

Now that you have an input definition, you can specify resolvers for your mutations. Create UDFs for each mutation using the following FQL:

create_firewall_rule

Query(
  Lambda(
    ["data"],
    Create(
      Collection("FirewallRule"),
      Var("data")
    )
  )
)

update_firewall_rule

Query(
  Lambda(
    ["id", "firewall_rule_input"],
    Update(
      Ref(Collection("FirewallRule"), Var("id")),
      { data: Var("firewall_rule_input") }
    )
  )
)

delete_firewall_rule

Query(
  Lambda(
    "id",
    Delete(
      Ref(Collection("FirewallRule"), Var("id"))
    )
  )
)

Add the following to your schema to specify resolvers for each mutation:

type Mutation {
  createFirewallRule(data: FirewallRuleInput): FirewallRule @resolver(name: "create_firewall_rule")
  updateFirewallRule(id: ID!, data: FirewallRuleInput!): FirewallRule @resolver(name: "update_firewall_rule")
  deleteFirewallRule(id: ID!): FirewallRule @resolver(name: "delete_firewall_rule")
}

Save your schema and replace the schema in the GraphQL tab of the Fauna dashboard. At this point, you have exactly the same functionality you started with. Because each query and mutation has an explicitly defined UDF as its resolver, you are now ready to perform your migration.

Final schema - migrate all at once

The final step in a migration is to update the logic in your UDFs, modify your type and input definitions in your schema, and upload your schema to Fauna. Because GraphQL is strongly typed, you must complete these actions in a single step. The simplest way to do this is by accepting some small amount of downtime for your application while you migrate, typically less than one minute. Techniques for migrating with zero downtime are beyond the scope of this post.

First, update the logic in your UDFs to handle the new shape of your data, reusing the same UDFs from the previous post.

Note: You do not have to update the create_firewall_rule UDF! Because GraphQL enforces the schema, any call to the createFirewallRule mutation already has the proper updated shape.

find_firewall_rule_by_id

Query(
  Lambda(
    ["id"],
    Call(
      "migrate_firewall_rule", 
      Ref(Collection("firewall_rules"), Var("id"))
    )
  )
)

update_firewall_rule_by_id

Query(
  Lambda(
    ["id", "new_rule"],
    Let(
      {
        ref: Ref(Collection("migrate_firewall_rule"), Var("id")),
        doc: Update(
          Var("ref"),
          Var("new_rule")
        )
      },
      Call("migrate_firewall_rule", Var("ref"))
    )        
  )
)

delete_firewall_rule

Query(
  Lambda(
    ["id"],
    Let(
      {
        ref: Ref(Collection("migrate_firewall_rule"), Var("id")),
        doc: Call("migrate_firewall_rule", Var("ref"))
      },
      Delete(Var("ref"))
    )
  )
)

Next, modify the type and input declarations in your GraphQL schema to reflect the new shape of your data:

type FirewallRule {
  action: String!
  port: Int!
  ipRange: [String]!
  description: String
}

input FirewallRuleInput {
  action: String!
  port: Int!
  ipRange: [String]!
  description: String
}

Finally, save your schema and replace the schema in the GraphQL tab of the Fauna dashboard. That's it!

Validating the migration

You can follow the steps laid out in the section Confirming zero defects in the previous post to verify that your migration was successful.

You can also use GraphQL queries and mutations to confirm that you observe the expected behavior. Navigate to the Collections tab in the Fauna dashboard and copy the id of an existing, unmigrated document in the FirewallRule collection. Return to the GraphQL tab and run the following query, replacing <FIREWALL_RULE_ID> with the id you copied:

query {
  findFirewallRuleByID(id:"<FIREWALL_RULE_ID>"){
    description
    ipRange
    port
    action
  }
}

The firewall rule should be displayed with an array value for ipRange. Return to the Collections tab and verify that the document now has an array value for ipRange.

Conclusion

When performing migrations with GraphQL, you apply the same principles that you apply when migrating with FQL. Before you perform your first GraphQL migration, you must specify input, type, and @resolver definitions in your GraphQL schema. Because GraphQL strongly enforces types via the schema definition, GraphQL migrations typically require some small amount of downtime as you replace your UDFs and schema.

This post leaves your underlying data unchanged until you access it, a pattern known as "just-in-time" (JIT) data migration. The final post in this series discusses JIT and two additional techniques for migrating your data and indexes with Fauna, along with sample code and guidance on choosing an appropriate approach for your application.

Migrating with user-defined functions

Rob Sutter — Fri, 03 Sep 2021 13:15:05 +0000

Modern applications frequently change as you deliver features and fixes to customers. In this series, you learn how to implement migrations in Fauna, the data API for modern applications.

The first post in this series introduces a high-level strategy for planning and running migrations. In this post, you learn how to implement migration patterns using user-defined functions (UDFs) written in the Fauna Query Language (FQL).

All of the code in this series is available on GitHub in Fauna Labs.

Migration scenario

Imagine you use an application to manage computers and appliances that communicate on your company's network. One of your domain objects is a firewall rule, which permits or denies inbound traffic to a resource on a given port from a specific range of IP addresses. The range of IP addresses is provided using CIDR notation and stored in your Fauna database as an FQL string

For this post, you have an updated user requirement to permit or deny inbound traffic on a particular port from an arbitrary number of IP address ranges. To satisfy this requirement, you must migrate the ipRange field type to an FQL array of strings. This demonstrates a common migration, changing a singleton to a collection as requirements evolve.

Pre-requisites

You do not need to install any additional software or tools. All examples in this post can be run in the web shell in the Fauna dashboard.

Create and populate your database

Create a new database in the Fauna dashboard. Do not select the "Pre-populate with demo data" checkbox.

Select the Collections tab and choose New collection. Enter firewall_rules as the Collection Name, leave the default settings for History Days and TTL, and choose Save to create your new collection.

Select the Shell tab (>_) to open the web shell. Copy and paste the following FQL into the editor window and choose Run Query to add three basic firewall rules.

Do(
    Create(Collection("firewall_rules"), { data: { action: "Allow", port: 80, ipRange: "0.0.0.0/0", description: "Universal HTTP access" } }),
    Create(Collection("firewall_rules"), { data: { action: "Allow", port: 443, ipRange: "0.0.0.0/0", description: "Universal HTTPS access" } }),
    Create(Collection("firewall_rules"), { data: { action: "Allow", port: 22, ipRange: "192.0.2.0/24", description: "Allow SSH from company headquarters" } })
)

Choose the Collections tab and select the firewall_rules collection. You should see three documents, each describing one of the previous firewall rules.

Encapsulating your data

In the first post in this series, you learn always to access your data via user-defined functions (UDFs). Before migrating your database, create UDFs that provide create, retrieve, update, and delete functionality.

Note: Do not abstract your UDFs to generic functions that wrap all operations across all collections! This approach introduces additional complexity and tightly couples your UDFs. Creating one UDF for each data access pattern aligns your UDFs with your business rules, making them easier to manage and more likely to be correct. This is especially true if you access your database via GraphQL, as you learn in the next post.

Create

Select the Functions tab in the Fauna dashboard and choose New function. Enter create_firewall_rule as the Function Name, leave the default value for Role, and paste the following FQL as the Function Body. Choose Save to create the UDF in your database.

Query(
    Lambda(
        "new_rule",
        Create(
            Collection("firewall_rules"),
            Var("new_rule")
        )
    )
)

This creates a UDF that accepts one parameter object, new_rule, and stores its value as a new document in the collection firewall_rules. This mirrors the format of the FQL Create() primitive, which accepts a collection name and a parameter object.

Passing a single object containing the required parameters is a best practice. Do not deconstruct the object into multiple parameters. Deconstructing can lead to incompatibilities, and future calls may fail if they do not provide the right parameters in the right order.

Test your UDF by selecting the Shell tab (>_) and pasting the following FQL query into the code editor. Choose Run Query to call your function and create a new firewall rule.

Call("create_firewall_rule", { data: { action: "Deny", port: 25, ipRange: "0.0.0.0/0", description: "Deny SMTP" } })

Return to the Collections tab and select the firewall_rules collection. You should now see four documents, including the new rule. Copy the id of the new rule for use in the next section.

Retrieve

Select the Functions tab in the Fauna dashboard and choose New function. This time, enter get_firewall_rule as the Function Name, leave the default value for Role, and paste the following FQL as the Function Body. Choose Save to create the UDF in your database.

Query(
    Lambda(
        ["id"],
        Get(
            Ref(Collection("FirewallRule"), Var("id"))
        )
    )
)

This creates a UDF that accepts one parameter object, id, and retrieves the entire associated document. Test your new UDF by selecting the Shell tab (>_) and pasting the following FQL query into the code editor, replacing with the id you copy from the output of running your create_firewall_rule function.

Call("get_firewall_rule", "<some_id>")

Choose Run Query to call your UDF. Your function should return the firewall rule you provided, along with its reference and timestamp.

Update

Return to the Functions tab and create another function named update_firewall_rule with the following FQL as the Function Body.

Query(
    Lambda(
        ["id", "new_rule"],
        Update(
            Ref(Collection("firewall_rules"), Var("id")),
            Var("new_rule")
        )
    )
)

This UDF differs slightly from the first two. It accepts two parameters, the id of the document to update, and a data object new_rule containing the fields to be updated.

Note: Updates in Fauna are not destructive. The fields of the provided document are merged with the existing document. To remove a field from a document, set the value of the field to null when updating the document.

You can prove this by testing your new UDF. In the web shell, paste the following FQL query, again replacing with the id you copy when creating your firewall rule.

Call("update_firewall_rule", ["<some_id>", { data: { action: "Allow", description: null } }])

Choose Run Query to call your UDF. Your function should return the updated document, with the action field modified to "Allow", the description field removed, and the port and ipRange fields unchanged.

Delete

Create another UDF named delete_firewall_rule from the Functions tab with the following FQL.

Query(
    Lambda(
        ["id"],
        Delete(
            Ref(Collection("firewall_rules"), Var("id"))
        )
    )
)

Test your UDF by pasting the following FQL in the web shell, replacing with the id you copy in a previous step.

Call("delete_firewall_rule", "<some_id>")

Choose Run Query to call your UDF, and Fauna removes the document from your database. You should have three documents in your firewall_rules collection and four UDFs - one for each CRUD primitive.

At this point, you should modify your client code to call the UDFs you create and avoid all calls to the FQL primitives. You have not changed any functionality in your application, but you are now prepared to make changes safely and perform migrations!

Migrating in steps

Migrating in steps reduces the risk of each stage of your migration.

Create a UDF that accepts a reference to a previous version of an object and updates it to the new format.
Modify the four UDFs you create in the previous section to call the migration function.
Create another UDF that verifies that your migration was successful. This is similar to a unit test of your migration UDF.
Remove any fields that you have deprecated with your migration.

Populating new field values

In the previous post, you learn that the first step in a migration is creating a UDF that populates the value of your new field from existing data according to your business logic. In this example, the business rule is to convert the string stored in ipRange to an array with one element, the existing value.

Create a UDF named "migrate_firewall_rule" with the following FQL code as the body.

Query(
    Lambda(
        ["firewall_rule_ref"],
        Let(
            { 
                doc: Get(Var("firewall_rule_ref")),
                ipRange: Select(["data", "ipRange"], Var("doc"))
            },
            If(
                IsArray(Var("ipRange")),
                Var("doc"),
                Update(
                    Ref(Var("firewall_rule_ref")),
                    { data: { ipRange: [Var("ipRange")] } }
                )
            )
        )
    )
)

The UDF accepts a reference, retrieves the specified document, and extracts the ipRange field. It determines whether the ipRange field is already an array. If so, it returns the unmodified document. If not, it updates the value of the ipRange field in the document to an array consisting of one element, the current value of ipRange. The IsArray() check makes this function idempotent. You always receive the same result no matter how many times you apply it to a document.

Note that this function accepts a Fauna reference as a parameter, not a string id. This function is invoked inside Fauna from other UDFs, so using the Fauna data type simplifies both writing and calling the function.

Updating your UDFs

After you save your migration UDF, you must update the UDFs you create in the previous section. The order of operations in each function differs based on the nature of the operation.

Create

You have two options when modifying your create_firewall_rule UDF. You can check the input format, make the value of ipRange an array if it is a string, and then create a document with the new info. However, this is awkward for large documents with nested objects, and duplicates the functionality you create with migrate_firewall_rule.

A simpler approach is to write the object as is and then apply the migration. Because Fauna is schemaless, you have the flexibility to perform both tasks in a single transaction, without restrictions on field types.

Replace the body of your create_firewall_rule UDF with the following FQL.

Query(
    Lambda(
        ["new_rule"],
        Let(
            { doc: Create(Collection("firewall_rules"), Var("new_rule")) },
            Call("migrate_firewall_rule", Select(["ref"], Var("doc"))
        )
    )
)

This function works even when the provided document is in the new format, because migrate_firewall_rule is idempotent. You can verify this by calling your UDF from the web shell.

First, invoke your function passing a string for ipRange.

Call("create_firewall_rule", {
  data: {
    action: "Deny",
    port: 25,
    ipRange: "0.0.0.0/0",
    description: "Unencrypted SMTP"
  }
})

Note that the returned value is the updated version of your document with an array as the value for ipRange, even though you provided a string. You can verify that the document is stored this way on the Collections tab.

Return to the Shell (>_) tab and invoke your function again, this time with an array for ipRange:

Call("create_firewall_rule", {
  data: {
    action: "Allow",
    port: 21,
    ipRange: ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"],
    description: "Allow FTP within private subnets"
  }
})

Your function succeeds, and stores your document exactly as provided.

Retrieve

Two characteristics of your migrate_firewall_rule UDF make adding the migration to your get_firewall_rule UDF simple.

The migrate_firewall_rule UDF is idempotent, so you can apply it to a document any number of times and get the same result.
The migrate_firewall_rule UDF returns the document in the updated format, and all UDFs return the value of the last statement in the function.

Taken together, this means that your get_firewall_rule UDF is written as a wrapper to your migrate_firewall_rule UDF, replacing the application-friendly id field with its associated Fauna-native reference.

Replace the body of your get_firewall_rule UDF with the following FQL.

Query(
    Lambda(
        ["id"],
        Call(
            "migrate_firewall_rule", 
            Ref(Collection("firewall_rules"), Var("id"))
        )
    )
)

Select the Collections tab and copy the id fields from two documents: one with a string value for ipRange and one with an array value for ipRange. Return to the web shell and invoke your get_firewall_rule twice, replacing and with their respective values.

Call("get_firewall_rule", "<string_id>")
Call("get_firewall_rule", "<array_id>")

Note again that each function call returns a document with an array value for ipRange. Return to the Collections tab and verify that both documents now have array values for ipRange.

Update

Updates in migrations work similarly to creates. First, make the provided changes to your document, then call the migration function to ensure the document is in the correct format.

Replace the body of your update_firewall_rule UDF with the following FQL.

Query(
    Lambda(
        ["id", "new_rule"],
        Let(
            {
                ref: Ref(Collection("migrate_firewall_rule"), Var("id")),
                doc: Update(
                    Var("ref"),
                    Var("new_rule")
                )
            },
            Call("migrate_firewall_rule", Var("ref"))
        )        
    )
)

Select the Collections tab and copy the id from a document with a string value for ipRange. Return to the web shell and invoke your update_firewall_rule function, replacing with the value you copied.

Call("update_firewall_rule", ["<some_id>", { data: { description: null } }])

This query removes the description field from the document while the migration UDF updates the value of ipRange to be an array.

Delete

Deletes are different from the other three operations. In order to migrate the data in a document during a delete, you must first apply the migration and then delete the document.

Replace the body of your delete_firewall_rule UDF with the following FQL.

Query(
    Lambda(
        ["id"],
        Let(
            {
                ref: Ref(Collection("migrate_firewall_rule"), Var("id")),
                doc: Call("migrate_firewall_rule", Var("ref"))
            },
            Delete(Var("ref"))
        )
    )
)

Select the Collections tab and copy the id fields from two documents: one with a string value for ipRange and one with an array value for ipRange. Return to the web shell and invoke your delete_firewall_rule twice, replacing and with their respective values.

Call("delete_firewall_rule", "<string_id>")
Call("delete_firewall_rule", "<array_id>")

Verify that both function calls return the document as it existed at the time of deletion with array values for ipRange.

Why should you migrate a document if you're deleting it anyway? If you do not access your database via GraphQL, a migration is strictly optional. However, it is considered a best practice to modify before delete since it provides a consistent return object shape and better supports temporality. Migrating delete functionality is required, however, if you access your database via GraphQL. The next post in this series addresses the reason for this requirement.

Confirming zero defects

In this example, you overwrite the existing field of a document with a new value rather than create a new field. How do you compare the new value to the previous value and ensure correctness in your migration?

Fauna provides temporality features that enable you to compare the history of a document at different points in time. Fauna retains thirty days of history data on your collections by default. This value can be increased or decreased both when a collection is created and at a later point in time.

The following FQL uses temporality to compare the values of ipRange in the same document at two points in time: the time of the last update, and one millisecond prior. It checks that the current value is an array, the previous value is a string, and the current value is equal to an array with one element, the previous value.

Note that this particular function only handles the happy path of a simple migration. It assumes that the tested document exists and was migrated in the last update without a change to the value of ipRange, only its format.

Create a UDF named "validate_migration" with the following FQL code as the body.

Query(
    Lambda(
        ["ref"],
        Let(
            { 
                new_doc: Get(Var("ref")),
                new_ts: Select(["ts"], Var("new_doc")),
                old_ts: Subtract(Var("new_ts"), 1),
                old_doc: At(Var("old_ts"), Get(Var("ref"))),
                new_ipRange: Select(["data", "ipRange"], Var("new_doc")),
                old_ipRange: Select(["data", "ipRange"], Var("old_doc"))
            },
            And(
                IsArray(Var("new_ipRange")),
                IsString(Var("old_ipRange")),
                Equals(
                    Var("new_ipRange"),
                    [Var("old_ipRange")]
                )
            )
        )
    )
)

Select the Collections tab and copy the id fields from two documents: one that you migrated from a string ipRange and one you created with an array ipRange. Return to the web shell and invoke your validate_migration twice, replacing and with their respective values.

Call("validate_migration", Ref(Collection("FirewallRule"), "<string_id>"))
Call("validate_migration", Ref(Collection("FirewallRule"), "<array_id>"))

Verify that calling with returns true and calling with returns false. You can also manually modify the value of a document and note its impact on the result of the validate_migration function.

Removing deprecated fields

Instead of migrating the value of ipRange, you could choose to create a new field ipRangeList and leave the existing field unmodified. In this case, the next step is to update all documents with a value for ipRange and set it to null, remove the field.

This doesn't apply to the current migration because you overwrite the existing field. More information on this topic is provided in the final post in this series.

Security and roles

Note that without using the migrate_firewall_rule UDF your retrieve function would need permission to write to your collection. This violates the separation of concerns and least privilege principles. By offloading migration responsibilities to a separate UDF, your business logic UDFs only need permission to call the migrate_firewall_rule.

Conclusion

UDFs are a powerful tool for decoupling your business logic from your migration logic. UDFs enable a number of additional techniques, including fine-grained access to resources and comparing different versions of documents at call time.

The next post in this series shows you how to perform migrations when you access Fauna via the Fauna GraphQL API. If you are not a GraphQL user, the final post in the series provides code and considerations for migrating your data and indexes.

Evolving the structure of your Fauna database

Rob Sutter — Thu, 02 Sep 2021 23:17:40 +0000

Modern applications change frequently as you deliver features and fixes to customers. These changes in data access patterns require changes to the structure and content of your data, even when using a schemaless database like Fauna.

In this series, you learn how to implement migrations in Fauna, the data API for modern applications. This post introduces a high-level strategy for planning and running migrations. In the second post, you learn how to implement the migration patterns using user-defined functions (UDFs) written in the Fauna Query Language (FQL). The third post presents special considerations for migrating your data when you access Fauna via the Fauna GraphQL API. In the final post, you learn strategies and patterns for migrating your data and explore the impact migrations have on your indexes.

Planning for migration

Planning is the most important element of a successful database migration. Preparing to evolve your data before you need to perform a migration reduces the risk of a migration and decreases the implementation time. Consider the four following key areas to prepare your database:

Encapsulate your data by limiting all access to your data to UDFs.
Migrate in steps to minimize the risk and allow for continuous uptime.
Choose a data update strategy based on your application's specific needs.
Assess the impact on your indexes to minimize performance impact.

Encapsulating your data

Always accessing your data via UDFs is generally a best practice with Fauna. Only allow clients to call specific UDFs, and prevent client-side calls to primitive operations like Create, Match, Update, and Delete. This separates your business logic from the presentation layer, which provides several benefits. UDFs can be unit-tested independently of the client, ensuring correctness. Because UDFs encapsulate your business rules, you can write them once and call them from different clients, platforms, and programming languages.

The following diagram represents a client side call to the FQL Create() primitive to create a document in a collection called notifications.

Suppose you receive a new requirement to add a field to each document that indicates whether a notification is urgent? You can modify the client code and publish a new version, but what about users who do not upgrade? What if the field is required to work with a downstream dependency? How do you handle existing documents and outdated clients? This single change can cascade and create additional complexity, which can lead to errors. That complexity multiplies when you combine multiple changes.

Instead, create a UDF, create_notification, and call that UDF directly from your client code with an FQL Call() statement.

With this change, newer versions of the UDF can accept calls from any version of the client. If the UDF expects a field and it is not provided, the UDF can set a reasonable default or calculated value and continue to completion. The second post in this series constructs a more complete example and shows how to test for existence of fields in UDF calls.

UDF-first development provides a number of other benefits, including support for attribute-based access control (ABAC), feature flags, versioning, and more. See this guide for additional information on UDFs.

Migrating in steps

Migrating in steps minimizes the risk of migrations and can allow for continuous uptime. Where possible, each step should change only one aspect of your database structure or data. A general pattern for updating the data type of a field has three steps: populating the values of the new field from existing data, confirming zero defects, and optionally removing any deprecated fields.

Populating new field values

The first step in a migration is creating a UDF that populates the value of your new field from existing data according to your business logic. The new value can be calculated using existing fields in the document, a calculated value, or other reasonable default.

Consider a field ipRange that stores a range of IP addresses in CIDR notation as an FQL string. You can create a new field ipRanges that stores the existing value as an FQL array without modifying the existing ipRange field.

You should call this UDF any time you access documents that have not yet been migrated, regardless of which data update strategy you choose.

Confirming zero defects

Next, create a second UDF that compares the value of your new field to the existing values. This UDF ensures that your data is migrated correctly. When you call this UDF varies based on your chosen data update strategy. Regardless of the strategy you choose, this UDF gives you the confidence that your migration succeeded on the document level. It also allows you to remove deprecated fields in the next step without worrying about data loss.

Removing deprecated fields

Removing deprecated fields is the final step in a migration. This step is strictly optional, but recommended. Removing deprecated fields reduces your data storage costs, particularly if those fields are indexed. It also removes unnecessary complexity from your data model. This is particularly helpful if you access Fauna via GraphQL, as you must explicitly define each field in your schema.

Choosing a data update strategy

You choose when to update your data, and how much of your data to update, based on your own use case and access patterns. There are three general data update strategies: just-in-time, immediate, and throttled batch updates.

Just-in-time updates

Just-in-time (JIT) updates wait until the first time a document is retrieved or altered to apply outstanding migrations. JIT updates check whether the specified document has been migrated and, if not, calls the UDF you specify before proceeding. The UDF-first pattern described in Encapsulating your data enables JIT updates without requiring the client to know a change has been made.

JIT updates are most appropriate for use cases where documents are accessed one at a time or in very small groups. They are especially good for documents with the optional ttl (time-to-live) field set, as these documents may age out of your database before they need to be migrated.

If you regularly retrieve many documents in a single query, JIT updates can degrade the performance of your query. In this case, you should choose between immediate and throttled batch updates.

Immediate updates

Immediate updates greedily apply your migration UDF to every matching document in your database in a single Fauna transaction. This simplifies future document retrieval, and maintains the performance of queries that return large data sets.

However, immediate updates require you to access and modify every document affected by your migration at once. If you infrequently access large portions of your data, this can create unnecessary costs. If you have indexes over the existing or new fields, these indexes must also be re-written along with your updated documents.

See Assessing the impact on your indexes for additional consideration for indexes and migrations.

Throttled batch updates

Throttled batch updates provide the benefits of JIT and immediate updates, but do so at the cost of additional complexity. Throttled batch updates use an external program to apply your migration UDF to groups (or "batches") of documents at a slower rate. You manipulate that rate by modifying either the number of documents in each batch or the period of time between each batch. This enables you to tune the time to completion, allowing you to migrate your entire data set without imposing any performance penalties.

If a request is made to access or alter a document that has not been migrated while your batch is still processing, you apply your migration UDF to that document first, just as you do with a JIT update.

Assessing the impact on your indexes

You cannot modify the terms or values of indexes, including binding objects, once they have been created. If you have an index over a previously existing field and want to use it for the newly migrated field, you must create a new index.

Fauna also limits concurrent index builds to 24 per account, not per database. Attempting to exceed this limit results in an HTTP 429 error that your application must handle. Additionally, index builds for collections with more than 128 documents are handled by a background task. This means that your transaction will complete successfully quickly, but you will not be able to query the index until it has finished building.

Conclusion

Successful data migrations depend heavily on planning. Use UDFs to encapsulate business logic, including performing any necessary data migrations. Break your migration in small steps with duplicated results to reduce risk at each stage. Finally, assess the impact on your indexes and develop a strategy for updating your data.

In the next post in this series, you learn how to implement the previous migration pattern using UDFs written in FQL.