DEV Community: Brice Pellé

AWS Amplify: How to interact with an existing S3 bucket

Brice Pellé — Fri, 09 Oct 2020 16:23:44 +0000

The AWS Amplify framework provides solutions that allows Frontend and Mobile web developers to easily implement solutions that interact with resources in the AWS cloud. For example, the Amplify CLI allows you to create a fully configured and secure S3 bucket to store items. Meanwhile, the Amplify Storage module lets you easily list the content of your bucket, upload items, and fetch items.

The Amplify libraries are not just great for interacting with resources created by the Amplify CLI. As highlighted here, you can manually configure your application to interact with an existing S3 bucket. In this article, I’m going to show you how you can set up an Amplify application to interact with an existing S3 bucket.

Let’s take a very common situation where you have an S3 bucket that was created outside an Amplify project, and that is used as an origin for a Amazon CloudFront distribution. The origin content is stored under a prefix in the bucket. Public access to the bucket is not allowed. Let’s set up an app that lets authenticated users manage files in the bucket.

Setting up your Amplify app

In your application folder, initialize Amplify and follow the prompt to configure.



$ amplify init

Initializing the app creates two IAM roles: AuthRole and UnauthRole. These roles are respectively used by authorized users and unauthorized users (if you configure guest access). You’ll use the AuthRole role in a bit.

Next set up auth to define a Cognito User Pool for your app. This will allow users to sign in.



$ amp add auth

Step through the configuration



 Do you want to use the default authentication and security configuration? 
❯ Default configuration

 How do you want users to be able to sign in? (Use arrow keys)
❯ Username

Select to configure additional settings. Set up greenlist filtering. This configures a Lambda function that checks the user’s email domain name when they try to register. Only emails with matching domains are permitted to register. You can edit the created Lambda function to implement custom checks and filtering.



 Do you want to configure advanced settings?
❯ Yes, I want to make some additional changes.

 Do you want to enable any of the following capabilities?
❯ Email Domain Filtering (whitelist)

 Do you want to enable any of the following capabilities? Email Domain Filtering (whitelist)
? Enter a comma-delimited list of allowed email domains (example: 'mydomain.com, myotherdomain.com'). your-own-domain.com

Create a policy resource

The next step is to create a policy granting access to the bucket that you will then associate with AuthRole. To do this, you will create a new policy category in your Amplify project. I like this approach because it allows to manage the policy independently of the auth category. Also your configuration is not lost if you update auth.

Open the file amplify/backend/backend-config.json, and define a new s3Auth resource in the policy category.



"policy": {
    "s3Auth": {
      "service": "iam",
      "providerPlugin": "awscloudformation"
    }
  }

Create the following file structure in the amplify/backend/ directory



policy
└── s3Auth
    ├── parameters.json
    └── template.json

Edit policy/s3Auth/parameters.json to make the AuthRoleName value available as a parameter to your template:



{
  "authRoleName": {
    "Ref": "AuthRoleName"
  }
}

Edit policy/s3Auth/template.json where you define the s3sappolicy inline policy for the AuthRole (see: "Roles": [{ "Ref": "authRoleName" }]). Note the “statement” and the “resource”. You should specify the actions allowed on the bucket and items. You can find some user policy examples here. The policy below is an example that limits access to files under <PREFIX>/.



{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "inline policy for s3 access",
  "Parameters": {
    "authRoleName": { "Type": "String" },
    "env": { "Type": "String" }
  },
  "Conditions": {
    "ShouldNotCreateEnvResources": {
      "Fn::Equals": [{ "Ref": "env" }, "NONE"]
    }
  },
  "Resources": {
    "s3sappolicy": {
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "s3-policy-for-bucket-access",
        "Roles": [{ "Ref": "authRoleName" }],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "InteractWithObjects",
              "Effect": "Allow",
              "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
              "Resource": "<BUCKET>/<PREFIX>/*"
            },
            {
              "Sid": "ListBucket",
              "Effect": "Allow",
              "Action": "s3:ListBucket",
              "Resource": "<BUCKET>",
              "Condition": {
                "StringLike": {
                  "s3:prefix": ["<PREFIX>", "<PREFIX>/*"]
                }
              }
            }
          ]
        }
      }
    }
  }
}

Run amplify env checkout <current-env-name> to populate the CLI runtime files and make it aware of the newly added custom resources; then run amplify push to deploy your changes.



$ amplify status

Current Environment: dev

| Category | Resource name              | Operation | Provider plugin   |
| -------- | -------------------------- | --------- | ----------------- |
| Policy   | s3Auth                     | No Change | awscloudformation |
| Function | s5b5b8cbf5b5b8cbfPreSignup | No Change | awscloudformation |
| Auth     | s5b5b8cbf5b5b8cbf          | No Change | awscloudformation |

Configuring your bucket

Your app will initiate cross-origin HTTP requests on your behalf to interact with your S3 bucket. You need to set up a CORS policy on your bucket to allow those requests.



<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <ExposeHeader>x-amz-server-side-encryption</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <ExposeHeader>ETag</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

You can set this up in the S3 console by selecting your bucket and opening the Permissions tab.

Connecting your app

At this point, you have created the resources necessary to authenticate users to your app and to access your S3 bucket. Now’s let’s connect your app. Your Amplify project provides an aws-exports.js file that has the configuration information for your Cognito User Pool. You need to specify the S3 setting for the Storage module.



import Amplify, { Storage } from 'aws-amplify'
import config from './aws-exports'

Amplify.configure(config)
Amplify.configure({
  Storage: {
    AWSS3: {
      bucket: '<BUCKET>',
      region: '<REGION>',
    },
    customPrefix: {
      private: '<PREFIX>/'
    },
  },
})
Storage.configure({ level: 'private' })

Note that there is no concept of public or protected resources for this application. The configuration defines a custom prefix for private level access, and the default access level is also set to private. You can list all files under in the bucket under <PREFIX>/



async () => {
  const result = await Storage.list('')
  console.log(result)
}

All uploads and downloads will be done at the private level as well.

Conclusion

Whether you are getting started on a brand new project or looking to interact with existing resources on AWS, Amplify provides tools that can help with that. In this article I showed you how to connect to an existing S3 bucket using the Amplify libraries in an Amplify project. To learn more about how Amplify can help your app dev project, visit the Amplify Framework Documentation.

Implementing pagination with AWS AppSync

Brice Pellé — Mon, 18 May 2020 20:45:04 +0000

Why Pagination

In this post, I'm going to show how you can get started with pagination in GraphQL using an AWS AppSync API and the AWS Amplify framework. The primary reason to use pagination is to control and limit the amount of data that is fetched from your backend and returned to your client at once. Pagination can help build efficient and cost-effective solutions by controlling the amount of work done to retrieve data in the backend. It can also improve overall responsiveness by returning smaller sets of data faster to the application.

Types of pagination

2 common forms of pagination are offset-based and token-based pagination. With offset-based pagination, you specify the page size and a starting point: the row after which, or the page at which you want to start fetching data. When using a page, the page along with the page size identifies the row after which to start fetching data (e.g.: offset = (page - 1) * page_size - 1 in a zero-based index). You can find offset-based pagination when dealing with relational databases. For example, in mysql you can fetch data from an offset using the LIMIT clause. In this example, 5 is the offset (fetch after that row) and 10 is the page size (return 10 items).



SELECT * FROM tbl LIMIT 5,10;  # Retrieve rows 6-15

With token-based pagination, a token is used to specify the record after which additional items should be fetched, along with the page size. The implementation of the token is system-specific. DynamoDB is an example of at system that uses token-pagination to paginate the results from Query operations. With DynamoDB, the result of a query may return a LastEvaluatedKey element. This is a token indicating that additional items can be fetched for this specific query. You can then continue the query and get the rest of the items by repeating the query and setting ExclusiveStartKey to the last value of LastEvaluatedKey.

How pagination works with AWS AppSync

AWS AppSync is a fully managed GraphQl service that makes it easy to build data-driven solutions in the cloud. Using the AWS Amplify GraphQL transform, you can quickly build AppSync APIs with types backed by data sources in your accounts. For example, you can use the @model directive in your schema to generate an API with types backed by DynamoDB tables.

Let’s take a look a how to work with pagination using Amplify and AppSync. I built a simple React app to showcase pagination with AppSync: Pagination with AWS AppSync. You can find the entire code here: https://github.com/onlybakam/todo-app-pagination. I am using the Amplify API library to easily interact with the AppSync API.

I created a new amplify project and created an AppSync API using the CLI. To find out how to get started with this, check out the Getting Started guide. I then created the following schema:



type Todo
  @model
  @key(
    fields: ["owner", "dueOn"]
    name: "ByDate"
    queryField: "listTodosByDate"
  ) {
  id: ID!
  name: String!
  description: String
  owner: String!
  dueOn: AWSDateTime!
}

The @key directive allows you to create a query to fetch todos per owner sorted by their due date. Check out Amplify Framework Docs - Data access patterns to find out more about how the @key can enable various data access patterns.

To fetch a list of todos for an owner, you execute the ListTodosByDate query. You can specify the amount of items you want returned using the limit argument. By default, the limit is set to 100. You can also specify the order the items are sorted by using sortDirection (set to ASC or DESC).



  query ListTodosByDate(
    $owner: String
    $dueOn: ModelStringKeyConditionInput
    $sortDirection: ModelSortDirection
    $filter: ModelTodoFilterInput
    $limit: Int
    $nextToken: String
  ) {
    listTodosByDate(
      owner: $owner
      dueOn: $dueOn
      sortDirection: $sortDirection
      filter: $filter
      limit: $limit
      nextToken: $nextToken
    ) {
      items {
        id
        name
        description
        owner
        dueOn
      }
      nextToken
    }
  }

The query returns a list of items and a nextToken field. If nextToken is set, this indicates there are more items to fetch. In a subsequent query, you can pass this value in the query arguments to continue fetching items starting after the final item that was last returned.

In the application, we want to be able to paginate forward and backwards through todos. To do this, we maintain 3 state variables



  const [nextToken, setNextToken] = useState(undefined)
  const [nextNextToken, setNextNextToken] = useState()
  const [previousTokens, setPreviousTokens] = useState([])

nextToken is the the token used to fetch the current items
nextNextToken is the token returned by the last fetch. If this token is set, you can paginate forward.
previousTokens is an array of previous tokens. These tokens allow us to paginate the todo list backwards. If there is a token in the array, you can paginate backwards.

A new set of todos is fetched whenever the owner, nextToken or sortDirection changes.



import { listTodosByDate } from './graphql/queries'
import { API, graphqlOperation } from '@aws-amplify/api'

useEffect(() => {
  const fetch = async () => {
    const variables = {
      nextToken,
      owner,
      limit,
      sortDirection,
    }
    const result = await API.graphql(graphqlOperation(listTodosByDate, variables))
    setNextNextToken(result.data.listTodosByDate.nextToken)
    setTodos(result.data.listTodosByDate.items)
  }

  fetch()
}, [nextToken, owner, sortDirection])

Loading the initial list of items

When the owner changes, all the fields are reset. nextToken is set to undefined which makes the query fetch items from the beginning. When the query returns, the value of nextToken in the result is assigned to nextNextToken. It’s important here to not immediately assign the value to the nextToken state as this would trigger another fetch right away.

Pagination forward

If nextNextToken is set, you can paginate forward. When the user presses the “Next” button, the current value of nextToken is pushed on the previousTokens array. Next, nextToken is set to the current value of nextNextToken. Finally nextNextToken is then set to undefined. When the query returns, again the value of nextToken in the result is assigned to nextNextToken. This process can be repeated as long as the query indicates that there are more items to paginate.

Pagination backwards

The previousTokens array stores the previously used tokens in order (think of is as a history stack). To paginate backwards, the last value is popped off the array and assigned to nextToken which triggers a new query. This allows you to repeat the query from a known "starting point". The query results may return a different nextToken. This is because items may have been inserted or deleted since the nextToken A was returned. By assigning the value of nextToken in the result is to nextNextToken, you keep paginating forward from the right position.

Conclusion

This post provided an overview of pagination and a simple solution for handling pagination in a React app with an AppSync API. Getting started with AWS AppSync and AWS Amplify is really easy. Check out the docs here.

You can find the code for this application here: https://github.com/onlybakam/todo-app-pagination. You can check out an implementation of it here: Pagination with AWS AppSync.