DEV Community: Robert J. Berger

Building AWS Ruby Lambdas that Require Gems with Native Extension

Robert J. Berger — Mon, 30 Jan 2023 01:29:45 +0000

What's the Problem?

If your Ruby development environment (e.g. a Mac) is different from your target Lambda in terms of operating system or architecture, there may be problems creating your Lambda image. This is especially true if your Ruby Gem Dependencies include C extensions. Without taking special steps, you may end up with a Lambda image that won't work when you try to run the Lambda.

This is due to the fact that some Gems have native extensions that ether have prebuilt versions for specific target OS and architectures. Many common Gems have native extentions. Some of these include Gems like nokogiri, json, nio4r and ffi.

When you run bundle install on your Mac to create a vendor/bundle that you expect to package into your Lambda image, you will only get the versions of these GEMS that are for your Mac OS and architecture. For instance an M1 Mac will install native extention Gems for arm64-darwin

If you zip that up into your Lambda function image and run it, you may see an error message similar to the following in your Cloudwatch logs when trying to run the Lambda (Gems that have Extensions will be listed in the Could not find line):

"errorMessage": "Could not find byebug-11.1.3, json-2.6.2, nokogiri-1.14.0, racc-1.6.2 in any of the sources",
"errorType": "Init<Bundler::GemNotFound>",

Note that this situation would happen not just for Macs, but for any development environment that is not the same OS (Linux) and runtime architecture (arm64 or amd64/x86_64) as your target Lambda.

Bundle in a Quick Lambda Linux Docker Container

The solution, as usual in such cases, is to use Docker to do the Gem Bundling in an environment that is the same as your target Lambda. It would be nice to still do it as a series of shell commands or part of a script and not bother with needing a Dockerfile. Turns out its pretty easy:

dir=`pwd`
bundle config set --local deployment 'true'
docker run --platform=linux/amd64 \
  -e BUNDLE_SILENCE_ROOT_WARNING=1 \
  -v `pwd`:`pwd` \
  -w `pwd` \ 
  amazon/aws-sam-cli-build-image-ruby2.7 bundle install
bundle config unset deployment
bundle config unset path

That command effectively runs bundle install in your current directory, except it is being run in the context of an AWS Lambda amd64 (x86_64) Linux with all the build tools for Ruby.

This Docker runtime has the current directory of your Mac mapped into its current directory. So it can read and write any files and directories it needs from your project.

Dissecting the Commands

Here's some more details on what each line of the command do:

dir=`pwd` Capture the current directory path in a shell variable just to make it easy to reuse in the following commands
bundle config set --local deployment 'true'
Set bundler configuration to deployment mode
- bundle install --deployment among other flags are depreciated and its now recommended to use bundle config to manage things like we do here.
  - Either style of command updates the local .bundle/config in your project with
```
 BUNDLE_DEPLOYMENT: "true"
 BUNDLE_PATH: "vendor/bundle"
```
  - --local ensures that this gets applied only to your project
  - deployment true:
    - Disallows changes to the Gemfile if the Gemfile.lock is older than Gemfile
    - It also sets the BUNDLE_PATH to vendor/bundle
    - This config be "sticky" after this which you probably don't want if you later do normal local development
      - Later config unset commands will unset these
- docker run
  The start of the docker command run. The following are the arguments to this command
- -e BUNDLE_SILENCE_ROOT_WARNING=1
  - Suppress the warning not to run as root, as docker default to the root user
  - Processes in docker container run as root unless you do some work to add users and set it to run as another user. And bundler issues warnings if you run it as root. So the -e flag creates an Environment Variable BUNDLE_SILENCE_ROOT_WARNING that is consumed by bundler and tells bundler not to issue that warning.
  - This is not required, but those warnings are aestheticly unpleasing.
- -v ${dir}:${dir}
  - Bind mount a volume: Maps the top of your ruby project with the Gemfile (`pwd` aka ${dir} aka your current directory) into the docker container with the same path.
- -w ${dir}
  - Working directory inside the container: Sets the working directory of the docker container runtime to be same as our current directory path.
  - This flag + the -v flag enables_the bundler process running in the container to read and write all the files and directories in your project.
- amazon/aws-sam-cli-build-image-ruby2.7
  - Positional argument that specifies the docker image to use
  - We use the amazon/aws-sam-cli-build-image-ruby2.7 image from DockerHub because its an image that has everything your ruby lambda Linux runtime will have plus it has all the Linux build tools needed to build any Ruby gems with extentions.
  - You don't need to be using SAM to use this image.
  - The same image is also available from AWS ECR as public.ecr.aws/sam/build-ruby2.7.
  - It would be possible to use other images that have what you need, but this one seems to work well.
- bundle install
  - Positional last argument: The bundle command to run once the docker image starts.
bundle config unset deployment
- This combined with the next command unsets what bundle config set deployment true set.
- You will probably want to run these two commands before doing any bundle commands before doing any normal local development.
bundle config unset path
- See previous command description

Zip it Up Good

You need to zip up your Ruby Project including the newly created vendor/bundle. This assume your current directory is your Ruby project with all your code, Gemfile, Gemfile.lock etc. Won't go into the details here. This is just the mechanics of packaging up your project into a zipfile suitable for uploading to your Lambda Function.

mkdir -p pkg/out
zip -q -r pkg/out/my-lambda.zip . -x pkg/*

Upload to Lambda function

This assumes you have already created your Lambda Function named my-lambda in your AWS Account in us-west-2.

aws lambda update-function-code --function-name my-lambda \
  --zip-file fileb://pkg/out/my-lambda.zip \
  --region us-west-2

For reference, the official docs: Deploy Ruby Lambda functions with .zip file archives (Ignore their instructions for using bundle)

Wrapping Things Up

If you use a modern Linux environment with the same processor architecture as your target Lambda, you can probably get away without having to use this Docker work around. Though it could still be safest to use this even there. The docker container being used is literally the same linux runtime as the target Lambda.

Otherwise, this technique should work for just about any development environment that can run Docker containers and run similar commands. Its only been tested it on a Mac though.

Do let me know if you have any feedback, know of any better ways or have improvements to this technique in the comments or by contacting me directly.

About the author

Robert J. Berger

As the Chief Architect, Rob guides the evolution of the InformedIQ Software and Infrastructure. His experience spans the rise and fall of many technology lifecycles from machine vision, digitization of professional video production equipment, Internet Infrastructure, Wireless, E-commerce, Big Data, IoT, DevOps and Machine Learning. He has been a founder or a technical leader in several startups in Silicon Valley.

Mastodon: https://hachyderm.io/@rberger

Python src Layout for AWS Lambdas

Robert J. Berger — Thu, 26 Jan 2023 08:41:39 +0000

Over the last several years, the Python community has been moving towards a packaging layout format known as the src layout as the recommended way to organize directories and files for Python Packages.

The main takeaways of the src layout are:

Your project has a src folder
- The src folder only has sub folder[s]. Each sub folder
- Is a Python Package
- Its name is the name of the package
- It contains an __init__.py (that's pretty much what makes it a Python Package)
- All Python source code that makes up your appplication (other than tests or build utilities) for the project are in these sub folder[s]
Modern Python tooling expects / works best with this layout
Makes packaging easier and less prone to errors
"editable" installs for development and regular installation for testing
Generally no need for "import trickery"
References at end of this article

We were looking for standards and best practices for organizing our Python AWS Lambda projects using Poetry in our monorepo and kept coming across the src layout as the recommended way to organize Python projects that had multiple modules. At the same time just about all examples of Python Lambda projects did not use src layout and in fact seemed to want to have the handler module at the top level of the project directory. There just weren't many examples of this combination of src layout Poetry and AWS Lambdas.

Turns out that it wasn't really difficult. The main "trick" is setting the Lambda handler name correctly.

All this example code is available at Informed/blogpost-python-src-layout

Example src layout for an AWS Lambda

Here's an example src layout suitable for AWS Lambda

This is a section of what could be a regular or monorepo that shows one lambda project: my_lambda
- You could have more lambda projects under services, each would be a separate subproject with the same layout
- How to implement such a multi-project monorepo is left to a future blog post
- The actual lambda code is in src/my_lambda
- There is an example of additional non-code files in src/my_lambda/stuff that might be used by the lambda or lambda layers such as an otel layer that will be part of the deployed zip image.
  - This is not required, just showing an example of how you can easily bundle in additional non-code files
- You can have additional Python modules in here like utils.py that are imported into the handler.py module
- If your Lambda application is more complex you can have local Packages as subdirectories of src/my_lambda

blogpost-python-src-layout
├── LICENSE
├── README.md
├── doc
│   └── python_src_layout_for_aws_lambdas.md
└── services
    └── my_lambda
        ├── CHANGELOG.md
        ├── LICENSE
        ├── README.md
        ├── poetry.lock
        ├── pyproject.toml
        ├── src
        │   └── my_lambda
        │       ├── __init__.py
        │       ├── handler.py
        │       ├── stuff
        │       │   ├── config.yml
        │       │   └── data.csv
        │       └── utils.py
        └── tests
            └── test_my_handler.py

Example Poetry pyproject.toml

[tool.poetry]
name = "my_lambda"
version = "0.0.0"
description = "This is my lambda which is mine"
authors = ["Da Dev <daDev@example.com>"]

[tool.poetry.dependencies]
python = "^3.9"
pyyaml = "6.0"

[tool.poetry.group.dev.dependencies]
coverage = {extras = ["toml"], version = "^6.5.0"}
pytest = "^7.2.0"
flake8 = "^6.0.0"
python-lambda-local = "^0.1.13"
boto3 = "1.20.32"

[tool.pytest.ini_options]
addopts = ["--import-mode=importlib"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry expects a src layout. The fact that the name is the same as the directory name under src and that directory is a Python Package (i.e. it has an __init__.py and source code) means that Poetry will treat everything in src/my_lambda as the package it's going to build. Everything in src/my_lambda and any dependencies under [tool.poetry.dependencies] will end up in the deployed zip image that will become the lambda function image. In this example we have the pyyaml package included.

The dependencies under [tool.poetry.group.dev.dependencies] will only be used for local builds, and not included in the zip image. You can put dependencies that are in the lambda runtime, like boto3 or any lambda layers (like Otel, or aws_powertools) in [tool.poetry.group.dev.dependencies] if you want them available for local testing.

There is also the option (and some folks recommend) to put a more up to date and fixed version of boto3 in [tool.poetry.dependencies] since AWS can change the version used in the runtime at any time. The main downside of this is it makes your uploaded image larger.

The section '[tool.pytest.ini_options]' is recommended for all new packages that use pytest for running their tests.

Set the Lambda Handler to be the "two dot" format

The main thing that is different for configuring Lambdas when you use the src layout is that you must specify the handler as

<package_name>.<handler_module_name>.<handler_function_name>`

Also known as the 2 dot solution that specifies the handler like a package import path.

In our example that would be:

my_lambda.handler.handler

Example src/my_lambda/handler.py

This is just a very minimal lambda function that demonstrates:

Importing the PyYaml package we specify as a dependency in the pyproject.toml
Printing the event info passed in the handler
Calling an imported function from the same lambda package (utils.my_util)
Fetching and printing one of the non-source files in src/my_lambda/stuff

import json
import os
import pkgutil
from . import utils

# We're not using pyyaml, just showing that it's installed
import yaml

print("Loading function")


def handler(event, context):
    print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event["key1"])
    print("value2 = " + event["key2"])
    print("value3 = " + event["key3"])

    print(utils.my_util())
    print(f"cwd: {os.getcwd()}")
    lines = pkgutil.get_data(__name__, "my_lambda/stuff/config.yml")
    print("File contents of my_lambda/stuff/config.yml:")
    print(lines)

    return event["key1"]  # Return the first key value

Building and deploying the Lambda

This is the basics of building and deploying the lambda using Poetry. You may have a more sophisticated CI/CD process.

These commands should be executed in the top level of the project (i.e. my_repo/services/my_lambda)

Building the distribution

Note: These commands may not work for your actual application if it has dependencies that need to be compiled as part of the build process, particularly if you are building your application on a machine that is not the same architecture as your target Lambda (i.e. building on an M1 Macintosh and targeting an x86_64 lambda). In that case you need to use Docker to do your building which is beyond the scope of this post.

These examples were only tested on an M1 Mac but should work on any modern Mac or *nix machine and will build zip images suitable for targeting Linux x86 Lambdas if you do not need to build binary dependencies.
The pip install shown later uses the argument --platform manylinux2014_x86_64 which forces the packaging to only include binary wheels built for Linux x86_64. The argument --only-binary :all: ensures that it will not try to compile any source only dependencies and instead will emit an error letting you know that you must build the package in the native target environment (i.e. use a Docker build process).

1) First we export the dependencies from Poetry so that we can later use pip install to create the files needed for the zip image.

poetry export -f requirements.txt --output requirements.txt  --without-hashes

2) Have Poetry build all the wheels and such

poetry build

This will result in a bunch of files in the directory dist the top level of your project.

3) Use pip to create the packages

poetry run pip install -r requirements.txt --upgrade --only-binary :all: --platform manylinux2014_x86_64 --target package dist/*.whl

This will generate all the wheels of all the dependencies in the package directory, suitable for zipping into the lambda image.

4) Zip up the image

cd package
mkdir -p out
zip -r -q out/my-lambda.zip . -x '*.pyc' out
cd ..

This will result in a file in my_repo/services/my_lambda/package/out/my-lambda.zip that is suitable for uploading to AWS as the Lambda Function image.

Create the Lambda function

You could do this in the AWS console or use the following AWS CLI Commands (If you use the AWS Console, it will create the execution role and trust policy automatically by default).

1) Create the execution role and trust policy

aws iam create-role --role-name my-lambda-ex --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'

You should get a result like the following.
Copy the arn from this output and you will use it for the next command

{
    "Role": {
        "Path": "/",
        "RoleName": "my-lambda-ex",
        "RoleId": "AROAWUWOOBVLDKY7ZE7P3",
        "Arn": "arn:aws:iam::1234567890123:role/my-lambda-ex",
        "CreateDate": "2023-01-12T06:08:22+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

2) Create the Lambda function

You have to set the handler to the 2 dot style my_lambda.handler.handler

Change the fake account-id (1234567890123) in the role arn to your AWS account-id

aws lambda create-function --function-name my-lambda \
  --zip-file fileb://package/out/my-lambda.zip \
  --handler my_lambda.handler.handler --runtime python3.9 \
  --role arn:aws:iam::1234567890123:role/my-lambda-ex

If you already have created the Lambda function

If you created the Lambda function some other way like via the Console or you want to update the Lambda function you created earlier, you can use the following command just to update it.

This mechanism loads the lambda from the S3 asset you uploaded earlier

aws lambda update-function-code --function-name my-lambda \
  --zip-file fileb://package/out/my-lambda.zip \
  --region us-west-2

Test the Lambda

At this point your lambda should be ready to test. You can use the default test input in the AWS Lambda Console:

{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}

You can run the default Test on the AWS Lambda console and you will see all the log output as well as the result which would look something like:

Test Event Name
basic

Response
"value1"

Function Logs
Loading function
START RequestId: 2f33f7b2-ddae-46df-a61b-1263ef404d6b Version: $LATEST
Received event: {
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
value1 = value1
value2 = value2
value3 = value3
Hello from my_util
cwd: /var/task
File contents of my_lambda/stuff/config.yml:
['thing:\n', '  hand: manicure\n']
END RequestId: 2f33f7b2-ddae-46df-a61b-1263ef404d6b
REPORT RequestId: 2f33f7b2-ddae-46df-a61b-1263ef404d6b  Duration: 1.57 ms   Billed Duration: 2 ms   Memory Size: 128 MB Max Memory Used: 40 MB  Init Duration: 172.83 ms

Or you could run it via the AWS CLI:

aws lambda invoke --cli-binary-format raw-in-base64-out \
  --function-name my-lambda \
  --payload '{ "key1": "value1", "key2": "value2", "key3": "value3"}' \
  outputfile.txt   --log-type Tail \
  --query 'LogResult' --output text |  base64 -d

This will print the log output (same as what was shown above in example of running the Test in the AWS Console)

You can see the returned value of the handler in outputfile.txt

Running tests with `Pytest`

There is also a simple test example in services/my_lambda/tests/test_my_handler.py

import pytest
import os
from my_lambda.handler import handler

event = {"key1": "value1", "key2": "value2", "key3": "value3"}
context = {}


def test_my_handler():
    # Emulate running in the same directory context as the lambda would
    os.chdir("src")
    assert handler(event, context) == "value1"

Before you run pytest for the first time in this project (or before you do any local development that needs any of the packages in the [tool.poetry.group.dev.dependencies] section of pyproject.toml) you need to run at least once:

poetry install

This will install all the dependencies listed in your main [tool.poetry.dependencies] and in [tool.poetry.group.dev.dependencies] in the virtualenv that is managed by Poetry.

You can run the test from services/my_lambda:

poetry run pytest

References on `src layout`

PyPA Python Packaging User Guide src layout vs flat layout
Pytest Good Integration Practices: Tests outside application code
The post that pretty much started the movement: Packaging a python library by Ionel Cristian Mărieș
Another early influential blog post Testing & Packaging by Hynek Schlawack
Organize Python code like a PRO
Convert a Poetry package to the src layout

Deploy EventCatalog to AWS CloudFront with Google SSO Access Control via Terraform

Robert J. Berger — Mon, 18 Jul 2022 05:15:01 +0000

This article shows how to deploy your own EventCatalog in AWS CloudFront via Terraform and updates to the Catalog via CI/CD (CircleCi in this case, but can be easily applied to other CI systems). It also shows how to use Lambda@Edge to implement Google SSO / OpenID Connect via the Widen/cloudfront-auth Project.

EventCatalog was created by David Boyne. It is an wonderful Open Source project that acts as a unifying Documentation tool for Event-Driven Architectures. It helps you document, visualize and keep on top of your Event Driven Architectures' events, schemas, producers, consumers and services.

You can go to the above links to find out more about EventCatalog itself.

Working on having a github repo with the full example at some point soon.

1. Create EventCatalog Project

1.1. Requirements

1.2. Generate the Scaffold Project Website
2. Create the Terraform to deploy to Cloudfront

2.3. Create the file main.tf

2.4. Create the lambda.tf file

2.5. Create the cloudfront.tf file

2.6. Create the variables.tf file

2.7. Create the sandbox.tfvars file

2.8. Create a placeholder lambda code zip file
3. Initial deployment with temp lambda@edge code
4. Build the Lambda@edge code with Widen/cloudfront-auth

4.9. Create the OAuth Credentials in the Google developers console

a. Create a new Project

b. Create OAuth Consent and Credentials

4.10. Generate the code for Lambda@edge
5. Deploy the EventCatalog content to S3

5.11. Manual deployment

5.12. Deployment with CircleCi
6. Deploy the new lambda@edge code with terraform
7. Improvements? Suggestions? Alternatives?
8. About the Author

Create EventCatalog Project

You can create a sample EventCatalog Project using the EventCatalog CLI. This can be the scaffolding for your own project. In this case we're going to use the sample project it will install as our project for this article.

Requirements

Node.js version >= 14 or above (which can be checked by running node -v). You can use nvm for managing multiple Node versions on a single machine installed
- We're going to be using Node.js version 16.x.x
Yarn version >= 1.5 (which can be checked by running yarn --version). Yarn is a performant package manager for JavaScript and replaces the npm client. It is not strictly necessary but highly encouraged.
- We're using Yarn 1.22.x

Generate the Scaffold Project Website

Go to a directory on your computer's filesystem where you want to save the project.
Generate the scaffolding for the project
- We're going to call the project my-catalog
```
npx @eventcatalog/create-eventcatalog@latest my-catalog
```

* This will generate a new directory structure as a git project:

```
my-catalog
├── services
│   ├── Basket Service
│   │     └──index.md
│   ├── Data Lake
│   │     └──index.md
│   ├── Payment Service
│   │     └──index.md
│   ├── Shipping Service
│   │     └──index.md
├── events
│   ├── AddedItemToCart
│   │     └──versioned
│   │     │  └──0.0.1
│   │     │     └──index.md
│   │     │     └──schema.json
│   │     └──index.md
│   │     └──schema.json
│   ├── OrderComplete
│   │     └──index.md
│   │     └──schema.json
│   ├── OrderConfirmed
│   │     └──index.md
│   │     └──schema.json
│   ├── OrderRequested
│   │     └──index.md
│   ├── PaymentProcessed
│   │     └──index.md
├── static
│   └── img
├── eventcatalog.config.js
├── .eventcatalog-core/
├── package.json
├── README.md
├── yarn.lock
├── Dockefile
├── .dockerignore
├── .gitignore
└── .git/
```

Change directory into my-catalog
You can preview the EventCatalog with the command:
```
  npm run dev
```
And then point your browser to http://localhost:3000
- You will be able to view the sample Events, Services, and Domains there.
Once you are done checking it out, kill the npm proces with CTL-C

Create the Terraform to deploy to Cloudfront

Create a terraform directory in my-catalog and add an assets directory to it
- You could make this directory outside of the catalog if you would prefer to manage it that way
```
mkdir -p terraform/assets
cd terraform
```
Create a .gitignore in the terraform directory

  curl https://raw.githubusercontent.com/github/gitignore/main/Terraform.gitignore -o terraform/.gitignore

Create the file `main.tf`

This file has all the terraform code to:

Set up the terraform environment
Specify the AWS provider
alt_fqdns a placeholder for now. May want to make alt_fqds a variable. It needs to be a list of strings Used by cloudfront.tf to specify aliases for the certificate and DNS but its kind of hard to support that with the sso callback

###
### Using locals to form variables by concatinating input variables
### Unfortunately can not do that in variables.tf or <env>.tfvars
###
locals {
  fqdn        = "${var.app_name}-${var.project_name}.${var.environment}.${var.base_domain_name}"
  alt_fqdns   = []
  zone_name   = "${var.environment}.${var.base_domain_name}"
  lambda_name = "${var.environment}-${var.project_name}-${var.app_name}-${var.lambda_name_suffix}"
}

terraform {
  required_version = ">= 1.2.0"
  required_providers {
    aws = {
      source = "hashicorp/aws"
      # Need to use version < 4.0.0 to work with cloudposse/cloudfront-s3-cdn
      version = ">= 3.75.2"
    }
  }
  # You should use a different state management than local
  backend "local" {}
}

provider "aws" {
  region  = var.region
  profile = var.profile
}

Create the `lambda.tf` file

Configure the AWS IAM role and policies for the lambda@edge
Create the lambda@edge service

### Set up IAM role and policies for the lambda
data "aws_iam_policy_document" "lambda_edge_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type = "Service"
      identifiers = [
        "lambda.amazonaws.com",
        "edgelambda.amazonaws.com"
      ]
    }
  }
}

# Define the IAM role for logging from the Lambda function.
data "aws_iam_policy_document" "lambda_edge_logging_policy" {
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["arn:aws:logs:*:*:*"]
  }
}

# Add IAM policy for logging to the iam role
resource "aws_iam_role_policy" "lambda_edge_logging" {
  name = "${local.lambda_name}-lambda_edge_logging"
  role = aws_iam_role.lambda_edge.id

  policy = data.aws_iam_policy_document.lambda_edge_logging_policy.json
}


# Create the iam role for the lambda function
resource "aws_iam_role" "lambda_edge" {
  name               = "${var.app_name}_lambda_edge_cloudfront"
  assume_role_policy = data.aws_iam_policy_document.lambda_edge_assume_role.json
}

# Create the lambda@edge function
resource "aws_lambda_function" "edge" {
  filename      = var.lambda_file_name
  function_name = local.lambda_name
  role          = aws_iam_role.lambda_edge.arn
  handler       = "index.handler"
  timeout       = "5"
  publish       = true

  # The filebase64sha256() function is available in Terraform 0.11.12 and later
  # For Terraform 0.11.11 and earlier, use the base64sha256() function and the file() function:
  # source_code_hash = "${base64sha256(file("lambda_function_payload.zip"))}"
  source_code_hash = filebase64sha256(var.lambda_file_name)

  runtime = "nodejs12.x"

}

Create the `cloudfront.tf` file

Create the CloudFront CDN instance and S3 bucket with lambda@edge association
- Uses the cloudposse/cloudfront-s3-cdn/aws terraform module to do all the hard work
- This module currently will work only with the hashicorp/aws provider of versions < 4.0.0
  - This is why we are not using the latest version of the hashicorp/aws provider
Create the TLS Certificate using AWS ACM

module "cloudfront-s3-cdn" {
  source  = "cloudposse/cloudfront-s3-cdn/aws"
  version = "0.82.4"

  namespace               = var.bucket_namespace
  environment             = var.environment
  stage                   = var.project_name
  name                    = var.app_name
  encryption_enabled      = true
  allow_ssl_requests_only = false
  # This will allow a complete deletion of the bucket and all of its contents
  origin_force_destroy = true

  # DNS Settings
  parent_zone_id      = var.zone_id
  acm_certificate_arn = module.acm_request_certificate.arn
  aliases             = concat([local.fqdn], local.alt_fqdns)
  ipv6_enabled        = true
  dns_alias_enabled   = true

  # Caching Settings
  default_ttl = 300
  compress    = true

  # Website settings
  website_enabled = true
  index_document  = "index.html"
  error_document  = "404.html"

  depends_on = [module.acm_request_certificate]

  # Link Lambda@Edge to the CloudFront distribution
  lambda_function_association = [{
    event_type   = "viewer-request"
    include_body = false
    lambda_arn   = aws_lambda_function.edge.qualified_arn
  }]
}


###
### Request an SSL certificate
###
module "acm_request_certificate" {
  source                            = "cloudposse/acm-request-certificate/aws"
  version                           = "0.16.0"
  domain_name                       = local.fqdn
  subject_alternative_names         = local.alt_fqdns
  process_domain_validation_options = true
  ttl                               = "300"
  wait_for_certificate_issued       = true
  zone_name                         = local.zone_name
}

Create the `variables.tf` file

Variable Definitions for EventCatalog-Sandbox

variable "region" {
  description = "The region to use for the Terraform run"
  default     = ""
}

variable "profile" {
  description = "The local IAM profile to use for the Terraform run"
  default     = ""
}

variable "environment" {
  description = "The environment to use for the Terraform run"
  default     = ""
}

variable "project_name" {
  description = "The name of the project to use"
  default     = ""
}

variable "app_name" {
  description = "The name of this app"
  default     = "eventcatalog"
}

variable "base_domain_name" {
  description = "The base domain name for the environment"
  default     = ""
}

variable "bucket_namespace" {
  description = "The namespace prefix for s3 buckets"
  default     = ""
}

variable "zone_id" {
  description = "The route53 zone id for the domain zone of the FQDNs"
  default     = ""
}

variable "lambda_file_name" {
  description = "The name of the lambda function file that was generated by the Widen/cloudfront-auth project"
  default     = ""
}

variable "lambda_name_suffix" {
  description = "The suffix to append to the lambda function name to make it unique if need to destroy and recrete CloudFront distribution"
  default     = "000"
}

Create the `sandbox.tfvars` file

This file sets or overrides the default values for the terraform run
- Set these as appropriate for your environment
- Region may need to be us-east-1

region           = "us-east-1"
profile          = "sandbox"
environment      = "rob"
project_name     = "blogpost"
app_name         = "eventcatalog"
lambda_file_name = "assets/temp.zip"

##
## These must be different for your environment
base_domain_name = "informediq-infra.com"
bucket_namespace = "informediq"
zone_id          = "Z10***********************K7U"

Create a placeholder lambda code zip file

We have a slight chicken and egg problem where we need to have the
Cloudformation name to create the lambda@edge code zip file with the
Widen/cloudfront-auth project.

So we'll make a dummy temp zip file to start with.

Create a file assets/temp.js with the following content:

  exports.handler = async (event) => {
      // TODO implement
      const response = {
          statusCode: 200,
          body: JSON.stringify('Hello from Lambda!'),
      };
      return response;
  };

Zip that file

  cd assets
  zip -r temp.zip temp.js
  cd ..

Initial deployment with temp lambda@edge code

Setup any credentials/login needed to run the AWS CLI / Terraform CLI from your shell window.
The first time you want to run things (or anytime you add terraform modules)
```
terraform init
```
Do the Terraform apply
- You could do a plan, but we're deploying for the first time anyway
- We are specifying it to use the sandbox.tfvars file to supply the input of the variables needed
```
terraform apply  -var-file=sandbox.tfvars
```
The first run of this may take a long time to complete. I've seen it seem to be stuck at
```
module.cloudfront-s3-cdn.module.logs.aws_s3_bucket.default[0]: Still creating...
module.cloudfront-s3-cdn.aws_s3_bucket.origin[0]: Still creating...
```
for more than 30 minutes. Not sure why. But after the first run its fast.

You may also get a warning:
```
  │ Warning: Argument is deprecated
```
You can ignore that. Seems to be something depreciated that is used by the cloudposse/cloudfront-s3-cdn/aws terraform module.
At the end of the run it will print out the outputs with something like:

  Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

  Outputs:

  cf_aliases = tolist([
    "eventcatalog-projectname.rob.informediq-infra.com",
    "eventcatalog.rob.informediq-infra.com",
  ])
  cf_domain_name = "d32pr*******z3r.cloudfront.net"
  s3_bucket = "informediq-rob-eventcatalog-origin"

Some of this info will be needed for the following steps to setup the Google SSO.

At this point if you tried to access https://eventcatalog.rob.informediq-infra.com you would get an error since the lambda@edge has the dummy code in it. This will be rectified in the following steps.

Build the Lambda@edge code with Widen/cloudfront-auth

Clone the Widen/cloudfront-auth repo in a directory outside of your my-catalog EventCatalog or terroform repo.

   git clone git@github.com:Widen/cloudfront-auth.git
   cd cloudfront-auth

Follow the instructions in the README for the Identity Provider of your choice. We are going to use the Google Hosted Domain mechanism:

Create the OAuth Credentials in the Google developers console

This assumes you don't already have a Project in the Google Developers Console but you have an account in the Google Developers Console.

Create a new Project

Click on the Projects pulldown on the very top menubar to the right of the Google Cloud logo
1. Click on New project in the modal popup that shows after clicking the pulldown
Fill in the New Project Form and click on CREATE

Create OAuth Consent and Credentials

Select APIs & Services from the menu bar on the left to go to that page of the project
Select Credentials from the new menu bar on the left
Click on Configure Consent Screen to configure the OAuth consent info
1. Select Internal and then click on CREATE
2. Fill in at least
  - App Name (EventCatalog Sandbox)
  - User Support email
    - This will be a pulldown and should have the email associated with the Google Dev account
  - Authorized domains
    - This should be the domain used for the email address of people logging in via Google SSO.
    - In my case this is informed.iq
  - Developer contact information email address
    - Can be your email
3. Click SAVE AND CONTINUE
4. Click SAVE AND CONTINUE on the next screen (Scopes Page)
5. Click on BACK TO DASHBOARD on the next screen (Summary Page)
6. Click on Credentials on the left hand nav bar to get back to the Credentials page
Click on + Create Credentials on the top menu bar and select OAuth client ID from the pulldown
1. Select Web application for the Application type
2. Under Authorized redirect URIs, enter your Cloudfront hostname with your preferred path value for the authorization callback. For our working example: https://eventcatalog-projectname.rob.informediq-infra.com/_callback
3. Click CREATE when done
Capture the resulting OAuth Client ID and Secret
- A modal window will show the OAuth Client ID and secret.
- You should store that somewhere, though you can also always view it on the Google Console later
- You can also download the JSON with the info and save it that way

We're now done with the Google Developer's Console

Generate the code for Lambda@edge

NOTE: Make sure you are in the Widen/cloudfront-auth directory for the following commands

Unfortunately, The Widen/cloudfront-auth project has not seen any updates in a while. But it is still widely used.

You can first run:

  npm audit fix --force

To at least remove some blatant high risk vulnerabilities. It seens to not impact the actual use of the project.

Execute ./build.sh. NPM will run to download dependencies and a RSA key will be generated.
- There will be some messages about the npm install
- There is a Warning that seems to be filling in the value of the first prompt >: Enter distribution name: you can ignore the warning and start filling in the values
  - Distribution Name - The value of cf_domain_name from the terraform run
  - Authentication methods - 1 Google
  - Client ID - The Client ID generated in the Google Console OAuth Credentials process
  - Client Secret - The Client Secret generated in the Google Console OAuth Credentials process
  - Redirect URI - The URL based on the domain name for the Cloudfront instance which was passed in the Google Console OAuth Credentials process
  - Hosted Domain - The email address domainname that will be used by people logging in via Google SSO
  - Session Duration - How many hours the session should last until the user needs to re-authenticate
  - Authorization methods - We are selecting 1 for Hosted Domain

NOTE: Redacting a few items for security

>: Enter distribution name: d32pr*******z3r.cloudfront.net
>: Authentication methods:
    (1) Google
    (2) Microsoft
    (3) GitHub
    (4) OKTA
    (5) Auth0
    (6) Centrify
    (7) OKTA Native

    Select an authentication method: 1
Generating public/private rsa key pair.
Your identification has been saved in ./distributions/d32pr*******z3r.cloudfront.net/id_rsa
Your public key has been saved in ./distributions/d32pr*******z3r.cloudfront.net/id_rsa.pub
The key fingerprint is:
SHA256:vJS0/*************************************************iE2ic rberger@tardis.local
The key's randomart image is:
+---[RSA 4096]----+
|       .o. =. .==|
|         oo.+.=.+|
|        ooo .o.B.|
|       o.+E...= .|
|        S .o   o.|
|       . +... + o|
|        o o+.o + |
|         . ==..  |
|          +=+o.  |
+----[SHA256]-----+
writing RSA key
>>: Client ID: 787***********************13cho.apps.googleusercontent.com
>>: Client Secret: GOCSPX-****************untA
>>: Redirect URI: https://eventcatalog-projectname.rob.informediq-infra.com/_callback
>>: Hosted Domain: informed.iq
>>: Session Duration (hours):  (0)  12
>>: Authorization methods:
   (1) Hosted Domain - verify email's domain matches that of the given hosted domain
   (2) HTTP Email Lookup - verify email exists in JSON array located at given HTTP endpoint
   (3) Google Groups Lookup - verify email exists in one of given Google Groups

   Select an authorization method: 1

Copy the resulting zip file found in the distribution folder in the Widen/cloudfront-auth directory to the assets directory in the terraform directory
- The process will output the path that the zip file was saved as relative to.
- In my setup the command to do the copy is:

  cp distributions/d32pr*******z3r.cloudfront.net/d32pr*******z3r.cloudfront.net.zip ../my-catalog/terraform/assets

Deploy the EventCatalog content to S3

You can deploy the content manually. But you really should use a CI/CD systems to deploy the EventCatalog content.

Manual deployment

The key actions needed are to:

Change directory to be in the top of the EventCatalog repo
Build the static assets using the EventCatalog cli
Copy the static assets to the S3 bucket created by Terraform

First we'll show doing it manually

Build the static assets
- Assumes you are in the top level of the EventCatalog Repo
- You only need to do yarn install the first time you use any of the commands
```
yarn install
yarn build
```
Upload the static assets to S3

* Assumes you have installed the AWS CLI 
* You have configured you local shell environment with proper IAM Profile to run the AWS CLI
* Use the actual s3 bucket you created in your terraform run
    * The example shows the bucket we've used in our working example

  aws s3 sync .eventcatalog-core/out s3://informediq-rob-eventcatalog-origin

Deployment with CircleCi

Assumes you have a CircleCI account and have it hooked up to your Github account.
- It is beyond the scope of this article to show how to setup and use Github and CircleCI
You will need to set CircleCi Project or Context environment variables:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_Region (Needs to be us-east-1)
Create the .circleci directory at the top of your EventCatalog repo directory
Create a file .circleci/config.yml with the following content
- You will need to substitue the s3 bucket name with the one you actually created with terraform

version: 2.1

# CircleCi Orbs (libraries) used by this config
orbs:
  node: circleci/node@5.0.2
  aws-s3: circleci/aws-s3@3.0.0

jobs:
  eventcatalog-contentbuild:
    docker:
      - image: cimg/node:16.15
    steps:
      - checkout

      - run:
          name: Install EventCatalog tooling
          working_directory: ~/project
          command: if [ ! -e "node_modules/@eventcatalog/core/bin/eventcatalog.js" ]; then yarn install; else echo "eventbridge seems to be cached"; fi;

      - run:
          name: Build the EventCatalog static content
          working_directory: ~/project
          command: |
            echo Running eventbridge build in `pwd`
            yarn build

      - aws-s3/sync:
          # Copy the static content to the S3 bucket
          # Replace the s3 bucket name with the one you actually created with terraform
          aws-region: AWS_REGION
          from: ~/project/.eventcatalog-core/out
          to: s3://informediq-rob-blogpost-eventcatalog-origin

workflows:
  eventcatalog-contentworkflow:
    jobs:
      - eventcatalog-contentbuild:
          context:
            # We're getting the AWS Credentials from our CircleCI Organization context
            # You could also just use Project level Environment Variables with
            # IAM AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
            - rberger-aws-user-creds

Once you have created this file and have all your commits in your EventCatalog Repo, push it to Github which should trigger your CircleCI run.

You can confirm that it sent it to s3 by using the AWS Console or CLI to view the contents of the S3 bucket.

Deploy the new lambda@edge code with terraform

Go back to your terraform directory.
- Make sure the new zip file is in the assets directory

Update the tfvars input file (sandbox.tfvars in our working example) with the new filename

lambda_file_name = "assets/d32pr*******z3r.cloudfront.net.zip"

region           = "us-east-1"
profile          = "sandbox"
environment      = "rob"
project_name     = "blogpost"
app_name         = "eventcatalog"
lambda_file_name = "assets/d32pr*******z3r.cloudfront.net.zip"
## On first run, set lambda_file_name to `assets/temp.zip`
# lambda_file_name = "assets/temp.zip"

##
## These should be different for your environment
base_domain_name = "informediq-infra.com"
bucket_namespace = "informediq"
zone_id          = "Z10***********************K7U"
##

Run terraform apply

terraform apply -var-file=sandbox.tfvars

A successful run will display the output values

* They should be something along the lines of the following:

```
Outputs:

cf_aliases = tolist([
  "eventcatalog-blogpost.rob.informediq-infra.com",
  "eventcatalog.rob.informediq-infra.com",
])
cf_domain_name = "d32pr*******z3r.cloudfront.net"
s3_bucket = "informediq-rob-eventcatalog-origin"
```

You should be able to go to ether of your cf_aliases.

For instance:

   https://eventcatalog.rob.informediq-infra.com

If you aren't already logged in, it should pass you to Google SSO authentication.
Once you are logged in you should see the Home Page of the EventCatalog

You can now start using the EventCatalog by updating the source files to fit your Domains, Services, and Events.

Improvements? Suggestions? Alternatives?

Please feel free to comment or contact me if you find any bugs, issues or have suggestions for improvements!

I am interested in hearing about alternatives to the Widen/cloudfront-auth as it has not been updated in a while.

About the Author

We're Hiring! https://informed.iq/careers/
Twitter: @rberger

This article was published originally in the InformedIQ Blog

Connect Quicksight to RDS in a private VPC

Robert J. Berger — Thu, 27 Jan 2022 08:45:26 +0000

There are cases where you want to connect AWS Quicksight to pull data from an RDS Database in one of your Private VPCs. Its one of those things that you don’t do often and its just funky enough and different enough from most AWS services that I have had to relearn how to do it each time. So here’s what I’ve learnt for posterity.

Quicksight can automatically connect to databases that can be accessed via a public IP. If your DB is publicly accessible to the Internet (with Security Group filtering of course), then you can pretty much ignore this article.

If you happen to have the weird case where your DB does have a public IP address but it is not actually accessible to the public Internet for policy, technical or historical reasons, then read on.

Quicksight `VPC Connection` Requirements

Quicksight has the option of creating a connection between you instance of Quicksight and one of your VPCs. It does that by injecting a Network Interface into a subnet you specify from the target VPC.

You only have to supply the

VPC ID that your target DB is in.
Subnet that is routable to the subnet your target DB is in
Security Group dedicated to the Quicksight connection that will allow all TCP taffic to the target DB
Optionally a DNS Inbound Endpoint

We're going to assume that the target DB and its VPC already exists.

You can use an existing Subnet as long as its in the same VPC and is routable to the subnets used by the target DB. By default subnets in a VPC can route to any other subnet in the VPC, but you should double check.

When you create the VPC Connection in the Quicksight management console, it will automatically create a Network Interface on the specified Subnet and will be associated with the Security Group specified.

Note that the Security Group associated with this new Quicksight Network Interface will be stateless. Any response packets coming back from a Quicksight request will have randomly allocated port numbers. Normally Security groups are stateful and handle this for you. But in the case of the Quicksight Network Interface you have to explicitly enable that any port is allowed for inbound.

The optional DNS Inbound Endpoint allows you to tell Quicksight to use the private DNS Resolver for your VPC instead of querying just the Public DNS zones. This is what is needed if your target DB has a Public IP address. Without this setting this Quicksight will get the Public IP address when it queries the Endpoint name of your DB. You will be scratching your head for days wondering why the connection is not working.

If you do use DNS Inbound Endpoint option, you will have to set it up in Route53.

Detailed instructions for all of this are described below.

A VPC Connection will allow Quicksight to connect to any of the following in your VPC:

Amazon OpenSearch Service
Amazon Redshift
Amazon Relational Database Service
Amazon Aurora
MariaDB
Microsoft SQL Server
MySQL
Oracle
PostgreSQL
Presto
Snowflake

You can reuse a VPC Connection for any Datasource in your Quicksight account in a region.

Subnet Info

Get VPC Info

We'll need the VPC ID and the CIDR Block associated

You can look at your RDS Configuration to see what VPC it is in.

In this example:

VPC ID ends in 2aed
CIDR Block: 10.0.0.0/16

Pick a Subnet for the Quicksight Network Interface to Use

The criteria are:

In the same VPC as the target DB
Is a private subnet
In the same Availability Zone as at least one of the subnets associated with the target DB
Routable to that subnet in the target DB
- Has a route table that routes to the VPC CIDR Block
- And the Target DB Subnets also can route to the VPC CIDR Block
Doesnt have an ACL that would block acces to/from the target DB
- This is the usual case

Find the subnets used on the target DB

In this example our target DB is an Aurora Postgres cluster. Looking at the RDS Console we can find the subnets its usings

Click on one of the subnets to view the subnet info

See what Availability Zone its in (us-east-1b in this example)

Confirm it routes to the VPC CIDR Block (10.0.0.0/16 in this example)

Go to the subnet view in the VPC Console.
Find an existing subnet that also routes to the same VPC CIDR Block (or overlapping subset with the DB subnet and its on the same Availability Zone)
- You could also create a new subnet for this as long as it meets the same criteria
In our example its the subnet that ends with 90dc

Security Group

Create a new Security Group dedicated to the Quicksight Network Interface. You don't have to create one specific to this, but it will make management easier than trying to mix it in with your existing Security Groups.

We'll call it Amazon-QuickSight-access. Nothing magic about the name though, whatever fits into your naming scheme.

Inbound Rules

Set the Inbound Rules to allow trafic on all TCP ports. As mentioned earlier, this is because this will be a stateless security group and all response packets will have random inbound ports.

Outbound Rules

The Outbound Rules should limit the destinations to just your target DB. The easiest way is to set the destination to be the Security Group set in your RDS Database.

You should also limit the outbound ports to be ones appropriate for your target DB, such as port 5432 for Postgres.

We ran into a problem where for historical reasons, there were some existing inbound rules in the Target DB that prevented us from using the Target DB as the destination security group, so we used a CIDR range that covered the Target DB range of addresses. This should be an unusual situation and you can probably ignore it.

DNS Resolver Endpoints (optional)

You only need to fill this in for cases where the DNS lookup of your Target DB Endpoint would be incorrect against Public DNS. The usual use case for this is when you have a somewhat complicated VPC Peering setup where your Target DB is on the other side of a VPC Peering setup. In that case, only the DNS Resolver in your private VPC may know the proper resolution of the Target DB Endpoint.

In our case, we had the unusal situation that our Target DB had a public IP address, so when Quicksight would do a DNS Query on the Target DB Endpoint name, it would get the Public IP address which was not valid for the VPC Connection. The workaround is for Quicksight to use the local VPC DNS Resolver. And thus our need to setup the DNS Resovler Endpoints

It took a while to figure out this was why we could never get the VPC Connection to work until we set this up. The diagnostics of the VPC Connection Validation check does not differentiate between Networking, DNS, or username/password problems. An issue in anyone of those can make the connection validation fail.

Route53 Resolver Inbound Endpoints

Create a Security Group for the Resolver

The resolver needs to have a security group for itself to allow the DNS requests to get to it.

Inbound Rules

Create a new Security Group and call it something like quicksight-route53-resolver or whatever fits your nameing scheme.
Set the Inbound Rules to allow for DNS UDP and DNS TCP from all sources on the VPC CIDR Block

Outbound Rules

Can leave the default outbound to all rule

Setup the Route53 Endpoint Resolver

You will need to go to the Route53 Console and select Resolver->Inbound endpoints and click on the Create Inbound Endpoint button.

Set the Endpoint name
- Something that fits your naming scheme
- Our example is quicksight-prod
Set the VPC ID to be the same as your VPC-ID used for your Target DB
Set the Security Group to the Security Group you created
Set the Availability Zone / Subnet for two IP Addresses for the resolver
- Could be any that are routable to the Subnet that is assigned to the VPC Connection.
- Should be a private subnet
- Might as well make one of them the same Subnet used by the VPC Connection
- Check the option Use an IP address that is selected automatically for both
- Click Submit when done

You will then end up with an Inbound Endpoint that will have been assigned two IP addresses. These addresses will be needed to supply to the VPC Connection and be used to update the Quicksight Security Group.

In this example the two IP Addresses are

10.0.100.120
10.0.101.80

Update the Quicksight Security Group for DNS

If you are using the DNS Resolver Inbound Endpoints feature, you will also have to update the Outbound Rules of the Security Group we created earlier for the Quicksight Network Interface. This is to enable Quicksight to be able to access the DNS Resolver as well as the Target DB.

To do this we will add DNS UDP and DNS TCP to the Output Rules of the Amazon-QuickSight-access Security Group for each of the two IP Addresses from the Inbound Resolver we just created. Note that you need to have the CIDR suffix /32 at the end when entering them into the Security Group editor.

Create the Actual VPC Connection

Now we have everything we need to setup the actual VPC Connection in the Quicksight management console.

You will of course need to have proper permissions to access and manage Quicksight in your account. That is beyond the scope of this article. We're going to assume you have all that already.

Enter the Quicksight Console, click on your username on the topr right of the page and select Manage Quicksight

Then select Manage VPC Connections in the left hand Navbar

Click on Add VPC Connection to create the new connection

Fill in the form with the info we found or created earlier:

VPC Connection Name
- Appropriate name of the connection based on your naming conventions
- Our example: my-aurora-db
VPC ID: The VPC ID we have been using earlier
- Our example ends with 2aed
Subnet ID: The Subnet we chose for the Quicksight Network Interface
- In our example it ends with 90dc
Security Group ID: The Security Group we created for Quicksight
- Our example: Amazon-QuickSight-access (ended in 16e8)
DNS Resolver Endpoints: The IP addresses from the DNS Resolver Inbound Endpoints
- Our example: 10.0.100.120 and 10.0.101.80

Create a Dataset using the VPC Connection

Now that the VPC Connection has been setup, we can use it to create a Dataset from the Target DB.

Click on the Quicksight logo on the top left of the screen to get back to the Quicksight home page.
Click on Datasets at the bottom of the left Navbar
Click on New dataset on the top right of the page.

Click on Aurora (or other source, but we're not going to show other sources in this article)

Fill in the `New Aurora data source` form

Data source name: Our example is my-data-source
Connection type: Select the VPC connection we created
- my-aurora-db

Database connector: PostgreSQL
Database server: The Endpoint of your Aurora DB
- This is the fully qualified DNS name of your DB endpoint
- You can find it in the Connectivity & security tab on the DB's RDS Console page
- You probably want to use a reader endpoint
Port: The proper port for your Target DB
- Postgres default is 5432
Database Name: The name of the database within the RDS of interest
- Same name you would use in a DB connection or in psql to connect to your working DB's
Username: The db username needed to connect
Password: The db password

Click on `Validate Connection`

It should turn to Validated with a checkmark if all went well. Should happen within a few seconds.

At this point you can now click on the Create data source button and do the normal Quicksight data source stuff. That is all independent of the VPC Connection and is not part of this article.

If the VPC Connection fails to Validate

If the Validate failed you are going to have to check several things. There will be an error message. You can click on the details link, but its probably not going to be helpful.

The error diagnosicts for the VPC Connection rarely gives you any more info other than it could not connect to the DB.

You need to determine if its because of:

The routing from the Quicksight Network Interface to the Target DB
The Security Group settings
Basic error in the regular connection parameters (Database name, Username, Password)
The Database Server is the correct value and Quicksight DNS query is getting the right value (private IPs not public or nothing at all)

Check for basic connectivity

You can check the basic connectivity (routing and security groups) is working using the Reachability Analyzer in the VPC Console. Unfortunately the analyzer has a limited set of elements that can be specified as a Source and Destination. The only one that applies to the Quicksight VPC Connection as a source and Aurora RDS as a Destination are Network Interfaces. So we're going to need to find the IDs of those two Network Interfaces.

Find the ID of the VPC Connection Network Interface

You will need to know the Network Interface ID of the interface created for the VPC Connection. To figure that out go to the EC2 Console page and click on Network Interfaces under Network & Security in the Navbar on the left

Then search for the name you used for the Quicksight connection. Our example was my-aurora-db It will be part of the description of the Network Interface associated with that connection. In our example it starts with eni-0b9e

Find the ID of the Target DB Network Interface

You will need to know any of the Network Interface IDs of the Target DB. There can be a few as there may be one per Availability Zone. It doesn't matter which one you choose.

Still on the EC2 Console page Network Interfaces page, search for the Target DB's Security Group name.
- You can find the Target DB Security Group name on the RDS Console page for your Target DB under the Connectivty & Security tab labeled VPC security groups
Select any one of them.
- Our example starts with: eni-050d

Run the Reachability Analyzer

Go to the VPC Console and click on the Create and analyze path button on the top right of the page

Give it a name
Select Network Interfaces for the Source type and Destination type
Specify the Network Interface ID of the Quicksight Network Interface we found
- Starts with eni-0b9e in our example
Specify the Network Interface ID of the Target DB Network Interface we found
- Starts with eni-050d in our example
Specify 5432 for the Destination port
- Or whatever port you set for your Target DB if not Postgres
Protcol is TCP

Click on Create and analyze path

If it all works you should see:

If that works, you configured the DNS Endpoint Resolver, but your VPC Connection / Dataset creation still doesn't work, you may want to repeat the Reachability Analyzer test for the DNS TCP and UDP ports in addition to the Postgres Port to double check for the DNS passing properly between the resolver and Quicksight.

Conclusion

If the Reachability Analyzer said connectivity is ok and it still doesn't work, then its probable that one of the other basic connection parameters is wrong, or there is something wrong with the Endpoint name. If you hadn't tried setting up the DNS Endpoint Resolver option, you can try that to see if there was a problem with how Quicksight was resolving the DNS for your Endpoint. That was what started this whole journey for me.

Otherwise, hopefully this did work for you and you can now happly view your Target DB in Quicksight!

How to Use Amplify Studio Figma Connector with Clojurescript

Robert J. Berger — Sun, 09 Jan 2022 21:56:44 +0000

Amplify Studio / Figma / Clojurescript / Reagent Tutorial

Implements the AWS Tutorial Build a Vacation Rental Site with Amplify Studio but instead of being Javascript based, uses Clojurescript for the project implementation. It does incorporate the Javascript output of Amplify Studio but all code to use it is in Clojurescript.

Tooling Used in the project:

Create Reagent App to create the project scaffold
Clojurescript (The whole point of this article!)
Shadow-CLJS as the build tool / compiler
Webpack Key to preping JSX and JS Files from Amplify Studio and UI Components to be used with shadlow-cljs transpiled clojurescript.
Babel Used by webpack to convert JSX to JS
Reagent (CLJS wrapper around React) for building your user interface
Amplify Studio and all the related AWS Amplify tooling
Figma AWS Amplify UI Kit

Prerequisites

Setup an Amplify Studio Project

All the initial setup of the Amplify Studio Project on AWS and the associated Figma project is already described in the first part of the excellent Build a Vacation Rental Site with Amplify Studio so will not repeat it here.

That first part of the article will have you do all the following in the appropriate Web Consoles and services AWS and Figma). You won't be doing any CLI commands on your local dev computer:

Setup an Amplify Studio project via the Amplify Sandbox
Create a basic data model in Amplify Studio
Deploy the start of the Amplify project to AWS
Create some sample data
Set up a Figma project using the Amplify UI Components and shows you how to modify it
Import the modified Figma project into the Amplify project
Link the data model and the UI Component in Amplify Studio
Create a collection view using Amplify Studio

Start by following the instructions from the original article, Build a Vacation Rental Site with Amplify Studio, up thru to the section: Pull to Studio
Once you have completed that, come back to here and follow the rest of this post at this point.

Creating an Amplify Studio App with Clojurescript

This is the actual instructions on how to create your Amplify Studio app in Clojurescript instead of Javascript. It replaces the remainder all the content after Pull to Studio from the original article Build a Vacation Rental Site with Amplify Studio.

Create a git repo with shadow-cljs / reagent scaffolding

Instead of using create-react-app that would have created a Javascript/React app, we’re going to use create-reagent-app to create the scaffolding of a shadow-cljs / reagent / react app repo.

In this tutorial, we will make this a git repo and snapshot the state at every stage so that if you make a mistake you can go back to an earlier step.

npx create-reagent-app  amplifystudio-cljs-tutorial
cd amplifystudio-cljs-tutorial
git init
git add -A
git commit -m "Initial Commit after create-reagent-app"
npm-install

Add webpack and related dependencies

Shadow-cljs can not directly consume JSX files that are the output of the Figma plugin and it needs some help to incorporate the AWS UI Components files that Amplify Studio injects into the project.

The use of Babel to prepare JSX files for Shadow-cljs is based on info from Shadow CLJS User’s Guide - JavaScript Dialects. This tutorial moves the babel management into webpack configuration as described later on.

The following dependencies are needed primarily to install webpack and its dependencies.

html-webpack-plugin and html-beautifier-webpack-plugin are used to inject the proper JS include for the output of webpack into the index.html.

npm i -D @babel/cli @babel/core @babel/preset-react @babel/preset-env babel-loader html-webpack-plugin html-beautifier-webpack-plugin process webpack webpack-cli

Then update any dependencies to the latest versions
If you don’t already have it, install npm-check-updates

npm install -g npm-check-updates

And then run it to update any dependencies to the latest versions ignoring specified versions in the package.json.

I like to start projects with the latest versions of everything. But you could just make sure shadow-cljs is the latest version, best to stay latest with that.

If you run it without the -u it will just show you what it would update and you could manually update the ones you care about.

ncu -u
npm install

Update your local git repo

git add -A
git commit -m "Snapshot after adding webpack dependencies"

If you want, you could push it to your own remote Github or other repository

Add AWS Dependencies

AWS Account
- If you don’t already have an AWS account, you’ll need to create one in order to follow the steps outlined in this tutorial. Create an AWS Account
Amplify CLI If you don’t already have the Amplify CLI installed you can install it with

npm install -g @aws-amplify/cli

Configure Account / IAM / CLI to work with Amplify If you already have an AWS account you want to use and you have things setup in your workstation / Terminal to use AWS CLI via profiles in ~/.aws/credentials, you can just set your profile in your terminal for the profile to use

export AWS_PROFILE=<your profile>

and you don’t need to do amplify configure.

If you haven’t set up aws amplify on your local dev machine before, follow the instructions at Configure the Amplify CLI

Install the aws-amplify libraries in your project

Still at the top of the amplifystudio-cljs-tutorial repo, install the libraries

npm i aws-amplify @aws-amplify/ui-react

You might want to commit the changes to git just as a snapshot in case the next step messes anything up.

git commit -a -m "After adding amplify deps"

Sync repo with Amplify project

Using the amplify CLI, pull the project info and ui-components into your repo.

You’ll get the command to do this from your Amplify Apps page that was created earlier.

If you are using an AWS account via IAM, you should log in to your AWS Console on your default browser. The following command is going to open up your default browser to authenticate to AWS.

If you are not using AWS IAM for auth, but are using the Amplify Console that has its own username/password style login, you don’t need to do anything in advance.

DON’T TYPE THIS EXACT LINE
Use the line from your environment as it has the appID for your application
The following line is just an example

amplify pull --appId dgt42342sdv765la --envName staging

This will eventually open a browser page to authenticate the process. As mentioned earlier, if you are using IAM for access, its easiest if you logged into the AWS Console with your browser first. If you forget to do this, you can still login now, and copy and past the link shown in the output of the CLI command and it will retry authenticating.

If you are using the Amplify Studio username/password, you will get that dialog on the browser and you can fill it in and click Yes

It will then prompt you for a bunch of things to set up your amplify project in this repo

Opening link: https://us-west-2.admin.amplifyapp.com/admin/dgt42342sdv765la/staging/verify/
✔ Successfully received Amplify Studio tokens.
Amplify AppID found: dgtkqevv765la. Amplify App name is: rental-cljs
Backend environment staging found in Amplify Console app: rental-cljs
? Choose your default editor:
  Android Studio
  Xcode (Mac OS only)
  Atom Editor
  Sublime Text
  IntelliJ IDEA
  Vim (via Terminal, Mac OS only)
❯ Emacs (via Terminal, Mac OS only)
(Move up and down to reveal more choices)

Of course the only choice that makes sense is Emacs 🤓
(Note even though it says via terminal, it works fine with GUI Emacs)

? Choose the type of app that you're building (Use arrow keys)
  android
  flutter
  ios
❯ javascript

Keep javascript

? What javascript framework are you using (Use arrow keys)
  angular
  ember
  ionic
❯ react
  react-native
  vue
  none

Keep react

? Source Directory Path:  src/amplify
? Distribution Directory Path: public
? Build Command:  npm run-script build
? Start Command: npm run-script start

Enter src/amplifyfor Source Directory Path
NOTE: we're going to send the amplify JSX/JS files to a directory that is NOT in the clojurescript classpath
Enter public for Distribution Directory Path
This build puts everything in public but other scaffolding or cljs projects may use some other path. It should be the same as the directory above js in the output-dir parameter in shadow-cljs.edn

You can keep the defaults for Build Command and Start Command

The rest of the config inputs and outputs:

✔ Synced UI components.
GraphQL schema compiled successfully.

Edit your schema at /Users/rberger/work/aws/amplifystudio-cljs-tutorial/amplify/backend/api/rentalcljs/schema.graphql or place .graphql files in a directory at /Users/rberger/work/aws/amplifystudio-cljs-tutorial/amplify/backend/api/rentalcljs/schema
Successfully generated models. Generated models can be found in /Users/rberger/work/aws/amplifystudio-cljs-tutorial/src/main
? Do you plan on modifying this backend? (Y/n) Y

Say Y for Do you plan on modifying this backend?

You might want to checkpoint your git repo again after this.

git add -A
git commit -m "After pulling Amplify Studio project"

You can make sure the basic reagent setup is still working by doing:

npm start

The first time you run this, it will take a while to download all the Clojurescript / Clojure dependencies.

And see that the app is running at http://localhost:3000
You will just see Create Reagent App on the page as a header.

Update to support mixing webpack with shadow-cljs

Based on David Vujic’s work Agile & Coding: Hey Webpack, Hey ClojureScript we’re going to add mechanisms to build the javascript code using webpack and the clojurescript code with shadow-cljs. This is necessary when using more recent versions of the AWS Amplify libraries.

Make sure Shadow-cljs dependencies are up to date

In shadow-cljs.edn make sure that the dependencies are up to date (you can check for the latest versions at Clojars)

 :dependencies
 [[reagent            "1.1.0"]
  [binaryage/devtools "1.0.4"]]

Shadow-cljs js-options

Add the following lines to shadow-cljs.edn between the :asset-path and :modules stanzas in the :app section as per Thomas Heller's article
How about webpack now?

   :js-options {:js-provider    :external
                :external-index "target/index.js"}

Make a template from index.html

Webpack will be used to update index.html with the proper script include that points to the webpack bundle.

Move `public/index.html` to `public/index.html.tmpl`

mv public/index.html public/index.html.tmpl

Edit `public/index.html.tmpl`

Add defer to the main script tag

Change:

<script src="/js/main.js"></script>

To:

<script defer src="/js/main.js"></script>

Add in a sytlesheet for the fonts

Add the following line after the other link tags in <head>

<link
  rel="stylesheet"
  href="https://fonts.googleapis.com/css?family=Inter:slnt,wght@-10..0,100..900&display=swap"
/>

Copy the Amplify CSS to public

Note that the source is styles.css (plural) and the destination is style.css (singular)

cp node_modules/@aws-amplify/ui/dist/styles.css public/css/style.css

Update the scaffold Clojurescript code to support Amplify

Here's where we actually get to the actually writing of some code to use the Amplify UI Components in an App.

Edit src/main/amplifystudio_cljs_tutorial/app/core.cljs with the following changes

Add the dependencies for the `require`

Add the aws amplify and ui imports to the require so it looks like:

Note that the amplify pull will populate src/amplify/ui-components and the webpack execution described further on, will set things up so the "ui-components/CardACollection" require can be fulfilled.

src/amplify should NOT be in the clojure[script] class path (usually set in shadow-cljs.edn :source-paths map)

(ns amplifystudio-cljs-tutorial.app.core
  (:require [reagent.dom :as rdom]
            ["/aws-exports" :default ^js aws-exports]
            ["aws-amplify" :refer [Amplify] :as amplify]
            ["@aws-amplify/ui-react" :refer [AmplifyProvider]]
            ["ui-components/RentalCollection" :default RentalCollection]))

Update the `app` function

This is the actual initial page code that is run by the render function. It is primarily hiccup syntax.

Hiccup describes HTML elements and user-defined components as a nested ClojureScript vector.

The first element is either a keyword or a symbol

If it is a keyword, the element is an HTML element where (name keyword) is the tag of the HTML element.

If it is a symbol, reagent will treat the vector as a component, as described in the next section.

If the second element is a map, it represents the attributes to the element. The attribute map may be omitted.

Any additional elements must either be Hiccup vectors representing child nodes or string literals representing child text nodes.

This code:

Displays an h1 header
Wraps the RentalCollection we created in Figma / ui-components with the AmplifyProvider

The :> is a function, adapt-react-class, that tells hiccup/reagent to interpret the next symbol as a React Component.
More info at: React Features in Reagent

The app function:

(defn app []
  [:> AmplifyProvider
  [:h1 "Amplify Studio Tutorial"]
   [:> RentalCollection]])

For comparison here is the equivalent Javascript:

function App() {
  return (
    <AmplifyProvider>
      <RentalCollection />
    </AmplifyProvider>
  );
}

Update the `main` function

This function is the first code called in the program. It is where you would put any initialization code and then it calls the render function that kicks of the reagent/react event loop.

Add a bit of logging so we can see that we're hitting the code at runtime
Add the Amplify initialization code.

(defn ^:export main []
  (js/console.log "main top")
  (-> Amplify (.configure aws-exports))
  (render))

In the Clojurescript statement:

(-> Amplify (.configure aws-exports))

-> is the thread-first macro. In this case it means that Amplify will be passed in as the second argument of the following form. I.E. its the equivalent to this Clojurescript statement:

(.configure Amplify aws-exports)

In ether case, it is the Javascript interop equivalent to:

Amplify.configure(config);

Setup Webpack / Babel

Babel config file

Babel does the work of converting JSX files to Javascript files suitable for consumption by webpack and shadow-cljs. It is called by webpack.

Create the file .babelrc in the top level of the repo with the content:

{
  "presets": ["@babel/preset-env", "@babel/preset-react"]
}

This tells babel to run the presets:

@babel/preset-env is a smart preset that allows you to use the latest JavaScript without needing to micromanage which syntax transforms (and optionally, browser polyfills) are needed by your target environment(s). This both makes your life easier and JavaScript bundles smaller!

and

@babel/preset-react loads the following plugins:

@babel/plugin-syntax-jsx - enables parsing of JSX
@babel/plugin-transform-react-jsx - transform JSX to Javascript
@babel/plugin-transform-react-display-name - Set displayName in the Javascript

And with the development option Classic runtime adds:

@babel/plugin-transform-react-jsx-self - sets self in the transformed code
@babel/plugin-transform-react-jsx-source - injects the source information (file, lineno) into the the Javascript

Webpack configuration file

Webpack:

At its core, webpack is a static module bundler for modern JavaScript applications. When webpack processes your application, it internally builds a dependency graph from one or more entry points and then combines every module your project needs into one or more bundles, which are static assets to serve your content from.

We are using it to convert the JSX ui-component files from Figma/Amplify Studio into vanilla Javascript via babel.

Webpack is also being used to bundle the src/amplify/models and src/amplify/ui-components directories/files that are pulled from amplify into the repo as modules so that their objects can be imported into the app. This is configured in the resolve block below.

This will be a webpack configuration file, webpack.config.js in the top level of the repo. The following will describe the elements we're going to use in that file.

Requires

The following requires the webpack modules and plugins used

const path = require("path");
const webpack = require("webpack");
const HtmlWebpackPlugin = require("html-webpack-plugin");
const HtmlBeautifierPlugin = require("html-beautifier-webpack-plugin");

Basic Webpack config

Set mode to development
entry - The file generated by shadow-cljs describing all the require/imports seen in the code
output - Where webpack should put its final bundle of javascript that will be included by a <script> tag in the index.html
devtool - Tells webpack to generate source maps to be consumed by the browser devtools

module.exports = {
  mode: "development",
  entry: "./target/index.js",
  output: {
    path: path.resolve(__dirname, "public"),
    filename: "js/libs/bundle.js",
    clean: false,
  },
  devtool: "source-map",

Rules

This is the main directives that tell weback what to do.

test: /\.m?js/, - Regex that specifies what file types to apply to the first rule to (ones that end with .mjs or .js)
- fullySpecified: false - the import / require statements should not end with file suffixes
- alias - Maps the path to the javascript files to a module name. This allows the code to require the Amplify Studio models and ui-components as importable modules.
test: /\.jsx$/ - Regex that specifies which file types to apply to the second rule (JSX files)
- exclude - Don't apply it to files installed by npm in /node_modules/
- use - Apply babel to the JSX files. The .babelrc file specified earlier tells babel to transform the JSX files to vanilla javascript

    rules: [
      {
        // docs: https://webpack.js.org/configuration/module/#resolvefullyspecified
        test: /\.m?js/,
        resolve: {
          fullySpecified: false,
          alias: {
            models: "../src/amplify/models/index.js",
            "ui-components": "../src/amplify/ui-components",
          },
        },
      },
      {
        test: /\.jsx$/,
        exclude: /node_modules/,
        use: ["babel-loader"],
      },
    ],

Plugins

This is where plugins are loaded.

process - This was needed as webpack 5 no longer includes a polyfil for the process Node.js variable. There were some dependencies that required process.env
HtmlWebpackPlugin - Enables creating public/index.html from a temlate so that webpack can inject the path to its bundle into the index.html. Also useful if you want to automate the updates of the index.html for other things.
HtmlBeautifierPlugin Cleans up the index.html with proper newlines mainly

  plugins: [
    new webpack.ProvidePlugin({
      process: "process/browser",
    }),
    new HtmlWebpackPlugin({
      template: "./public/index.html.tmpl",
      filename: "index.html",
    }),
    new HtmlBeautifierPlugin(),
  ],

The Html plugins / index.html templating are not totally necessary. You could just add your own script tag to index.html instead such as:

<script defer src="js/libs/bundle.js"></script>

The full `webpack.config.js`

Create a file webpack.config.js also at the top level of the repo with the content:

const path = require("path");
const webpack = require("webpack");
const HtmlWebpackPlugin = require("html-webpack-plugin");
const HtmlBeautifierPlugin = require("html-beautifier-webpack-plugin");

module.exports = {
  mode: "development",
  entry: "./target/index.js",
  output: {
    path: path.resolve(__dirname, "public"),
    filename: "js/libs/bundle.js",
    clean: false,
  },
  devtool: "source-map",
  module: {
    rules: [
      {
        // docs: https://webpack.js.org/configuration/module/#resolvefullyspecified
        test: /\.m?js/,
        resolve: {
          fullySpecified: false,
          alias: {
            models: "../src/amplify/models/index.js",
            "ui-components": "../src/amplify/ui-components",
          },
        },
      },
      {
        test: /\.jsx$/,
        exclude: /node_modules/,
        use: ["babel-loader"],
      },
    ],
  },
  resolve: {
    extensions: ["", ".js", ".jsx"],
  },
  plugins: [
    new webpack.ProvidePlugin({
      process: "process/browser",
    }),
    new HtmlWebpackPlugin({
      template: "./public/index.html.tmpl",
      filename: "index.html",
    }),
    new HtmlBeautifierPlugin(),
  ],
};

Add a script to run webpack

Add the following line to the ”scripts” section of package.json. It will allow you to run a that will update the bundle automatically when you change any of the amplify files or when shadow-cljs updates the target/index.js

    "pack": "webpack --watch"

Update git

Update .gitignore

add the following to .gitignore

/target/

Add all the new files to the commit

git add -A
git commit -m "Sync up all the final changes"

Running the development service locally

Start the shadow-cljs watch process. (shadow-cljs watch app) using the npm command:

npm start

And in another terminal window, also at the top of the repo run the webpack watch process:

npm run pack

Should see something like the following. The images and values are dependent on how you set up the data when following along with the first part of the Build a Vacation Rental Site with Amplify Studio.

Troubleshooting

If when you start the shadow-cljs process, npm start, and you get something like:

...
shadow-cljs - watching build :app
[:app] Configuring build.
[:app] Compiling ...
[2022-01-02 21:55:15.214 - WARNING] :shadow.cljs.devtools.server.util/handle-ex - {:msg {:type :start-autobuild}}
AssertionError Assert failed: (map? rc)
...

The js-provider :external config in shadow-cljs.edn is masking the actual error. In order to see what the error is, comment out the :js-options block in shadow-cljs.edn like:

:builds
 {:app
  {:target     :browser
   :output-dir "public/js"
   :asset-path "/js"
   ;; :js-options {:js-provider    :external
   ;;              :external-index "target/index.js"}
   :modules    {:main
                {:init-fn amplifystudio-cljs-tutorial.app.core/main}}}

and then run npm start again and see what the error is. Correct the error and then remember to uncomment the :js-options block.

Completing the Tutorial with Amplify UI Overrides

We pick back up the original tutorial at the Use a Prop section to show how the UI Components can be customized just with Component Props and runtime Overrides.

Overrides are a powerful feature that are builtin to the Amplify UI Components and allow you to inject attributes into the children of components at runtime. It makes the Amplify UI Components very flexible without having to modify the actual code of the components. This allows you to update the Figma design aspects and still update your local copy of the ui-components with an amplify pull since you don't make local changes to that code.

Use a Prop

You can customize these React components in your own code. First, you can use props in order to modify your components. If you wanted to make your grid of rentals into a list, for example, you could pass the prop type="list" to your RentalCollection.

In Javascript you would say:

<RentalCollection type="list" />

and in Clojurescript:

[:> RentalCollection {:type "list"}]

And that will make the view go from a grid to a list:

The props are listed for each component type at Amplify UI Connected Components

Use an Override

Overrides allow you to inject props into the children of a component.

In our example RentalCollection, the images in the child cards are kind of squashed. To fix that we want to set the objectFit prop of the image element of the card to cover.

In Javascript you would use:

<RentalCollection
  type="list"
  overrides={{
    "Collection.CardA[0]": {
      overrides: {
        "Flex.Image[0]": { objectFit: "cover" },
      },
    },
  }}
/>

In Clojurescript we use:

[:> RentalCollection {:type "list"
                      :overrides {"Collection.CardA[0]"
                                  {:overrides {"Flex.Image[0]"
                                               {:object-fit "cover"}}}}}]])

Now the images are no longer squished:

Themes and Conclusion

That completes showing the differences of using Clojurescript instead of Javascript with Amplify Studio and Amplify UI Connected Components.

You can refer back to the original AWS Tutorial Build a Vacation Rental Site with Amplify Studio for the remaining content on how to use the AWS Amplify Theme Editor in Figma to add a theme to the UI Components. This should work without having to change any of your Clojurescript code as you modify the Ui component code that you load via amplify pull via Figma.

It is also possible to apply themes directly in your code. Doing that with Clojurescript will be left to a possible future article.

The full project / code for this repo is at

rberger / amplifystudio-cljs-tutorial

Tutorial on using Clojurescript with Amplify Studio / Figma

Feel free to post issues or questions there.

Set up SSL/TLS for shadow-cljs https server

Robert J. Berger — Thu, 21 Oct 2021 06:10:50 +0000

While developing clojurescript web apps, you may require that the development http server shadow-cljs operate with SSL/TLS to serve up HTTPS, not just HTTP.

This is particuarly true if you need to test things out on an iPhone or Android phone but still run with the development server so you can iterated changes just as quick as when you are working with desktop clients.

Its a bit tricky to get everything lined up to make SSL/TLS work locally as Apple (and I presume other browsers) no longer support self-signed certificates for HTTPS servers. So you need a private CA and a certificate generated from the private CA.

This is a guide to set up:

A Private Certificate Authority (CA)
A Server Certificate for your shadow-cljs development server
How to configure shadow-cljs.edn for SSL
How to install the CA Root Certificate on other clients (like an iPhone) so they can access the shadow-cljs servers

**NOTE: This server / CA / Certificates should never be used in production or in any particularly public way. It’s not secure. We’re doing this to get around the normal browser / server security just for local development.

Install mkcert

See the following for more info or how to install on Linux: GitHub - FiloSottile/mkcert

Install mkcert on macOS

> brew install mkcert
> brew install nss # if you use Firefox

Create a local CA to be used by mkcert and clients

> mkcert -install
Created a new local CA 💥
Sudo password:
The local CA is now installed in the system trust store! ⚡️
The local CA is now installed in the Firefox trust store (requires browser restart)! 🦊
The local CA is now installed in Java's trust store! ☕️

Create a pkcs12 certificate

Easiest to do this in the directory you are running the shadow-cljs project.
Create a subdirectory ssl at the same level as shadow-cljs (top level of the repo usually) and cd into ssl

❯ cd ~/work/my-project
❯ ls
Makefile        RELEASE_TAG     bin             dev             package.json    shadow-cljs.edn test
README.org      amplify         deps.edn        node_modules    resources       src             yarn.lock

❯ mkdir ssl
❯ cd ssl

Create the certificate that the shadow-cljs servers will use as their server certificates. You want to specify all the domains and IPs that would be associated with the certificate and the way you will access the server.
In my case my iMac has two interfaces plus localhost. One interface is the Ethernet, the other is the wifi. And just to be safe, I’m putting in their IPv6 addresses as well.

❯ mkcert -pkcs12 discovery.local localhost  192.168.20.10 192.168.20.11 127.0.0.1 ::1 fd95:cb6f:7955:0:1878:b8b5:1b3b:ad27 fd95:cb6f:7955:0:4cd:c922:d1b3:2eb5

Created a new certificate valid for the following names 📜
 - "discovery.local"
 - "localhost"
 - "192.168.20.10"
 - "192.168.20.11"
 - "127.0.0.1"
 - "::1"
 - "fd95:cb6f:7955:0:1878:b8b5:1b3b:ad27"
 - "fd95:cb6f:7955:0:4cd:c922:d1b3:2eb5"

The PKCS#12 bundle is at "./discovery.local+7.p12" ✅

The legacy PKCS#12 encryption password is the often hardcoded default "changeit" ℹ️

It will expire on 20 January 2024 🗓

Install the cert into the keystore

NOTE: The passwords you use here should not be used anywhere else, particularly on public services. They do not have to be super secret, great passwords as they will be in the clear in your shadow-cljs.

You will create a local Java JKS Keystore in ssl to be used by shadow-cljs servers

Destination Password: This will be the password specified in shadow-cljs.edn to gain access to the keystore. Our example will be super-secret
Source keystore password: The password that mkcert used to generate the Server Certificate and thus the password of the Server Certificate. I could not find a way to specify it. It defaults to changeit

❯ keytool -importkeystore -destkeystore keystore.jks -srcstoretype PKCS12 -srckeystore discovery.local+7.p12

Importing keystore discovery.local+7.p12 to keystore.jks...
Enter destination keystore password: super-secret
Re-enter new password: super-secret
Enter source keystore password: changeit
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Configure shadow-cljs.edn to enable SSL

Mainly need to add an :ssl coda to the start of the shadow-cljs.edn

{:deps  true
 :nrepl {:port 8777}
 :ssl {:keystore "ssl/keystore.jks"
       :password "retold-fever"}
 :dev-http {8020 {:root "resources/public"}}
 ... rest of your shadow-cljs.edn file...

No need to specify the hostnames. In fact that will limit access to IP addresses that resolve to that name which may be incorrect.
More info on the :ssl configuration at Shadow CLJS User’s Guide: SSL

[Re]start your shadow-cljs watch process and it should say something like the following at some point in its startup where https is the protocol shown for the http and shadow-cljs servers:

...
shadow-cljs - HTTP server available at https://localhost:8020
shadow-cljs - server version: 2.15.8 running at https://localhost:9631
shadow-cljs - nREPL server started on port 8777
shadow-cljs - watching build :app
...

Assuming you set the certificate to support any other domain names and IP addresses associated with your computer running this, they will also work as the host address in your client URL accessing this server. But only if running on the same machine as this server.

If you want to make another device (like an iPhone or another computer) access this server, follow the next steps.

Export the Root CA of your Private CA to other Clients

In order for other machines on your LAN to access the shadow-cljs server running with the Private CA and Server certificate set up in the earlier steps, you will need to export the Root CA from that machine to these other clients.

Find the location of the Root Certificate of the Private CA

When you ran mkcert install it created the root certificates of the Private CA and stashed them somewhere appropriate for your system. You can find out where with the command:

❯ mkcert -CAROOT
/Users/rberger/Library/Application Support/mkcert

❯ ls '/Users/rberger/Library/Application Support/mkcert'
rootCA-key.pem rootCA.pem

You will want to copy the rootCA.pem to other clients that would access the shadow-cljs servers.

For transferring to other Macs or iOS devices

open '/Users/rberger/Library/Application Support/mkcert'

Which will open a finder window with the directory where these pem files are:

And then select AirDrop to send them to other macOS or iOS devices
Otherwise you can email it or send the file some other way to a destination device.

Install the Private CA Root Cert on iOS device

Once you send the Cert to an iOS device, you will get a message
Select iPhone and then select Close:
Go to Settings and you’ll see the a new option Profile Downloaded Click on that and the go thru the rest of the dialogs agreeing to Install the downloaded profile.

After completing all the install dialogs, this client should be ready to connect to the shadow-cljs using https.

Accessing AppSync APIs that require Cognito Login outside of Amplify

Robert J. Berger — Wed, 01 Sep 2021 01:23:37 +0000

The Need

You have this great Amplify App using AppSync GraphQL. You eventually find that you need to be able to access that data in your AppSync GraphQL database from tools other than your Amplify App. Its easy if you just have your AppSync API protected just by an API Key. But that isn't great security for your data!

One way to protect your AppSync data is to use Cognito Identity Pools. Amplify makes it pretty transparent if you are using Amplify to build your clients. AppSync lets you do really nice table and record level access control based on logins and roles.

What happens if you want to access that data from something other than an Amplify based client? How do you "login" and get the JWT credentials you need to access your AppSync APIs?

Use AWS CLI

The most general way is to use the AWS CLI to effectively login and retrieve the JWT credentials that can then be passed in the headers of any requests you make to your AppSync APIs.

Unfortunately its not as easy as just having your login and password. It also depends on how you configured your Cognito Identity Pool and its related Client Apps.

Cognito User Pool Client App

You can have multiple Client Apps specified for your Cognito User Pool. I suggest having one dedicated to these external applications. That way you can have custom configuration just for this and not disrupt your main Amplify apps. Also you can easily turn it off if you need too.

In my case I created a new client app shoppabdbe800b-rob-test2 as a way to test a client app with no App Client Secret. This makes it easier to access from the command line as you do not have to generate a Secret Hash (will describe how to deal with that below).

If you want to allow admin level access (ie a user with admin permission) you need to check Enable username password auth for admin APIs for authentication (ALLOW_ADMIN_USER_PASSWORD_AUTH)

If you want to allow regular users to login you must also select Enable username password based authentication (ALLOW_USER_PASSWORD_AUTH)

The defaults for the other fields should be ok. Be sure to save your changes.

Minimal IAM permissions

As far as I can tell, these are the minimal IAM permissions to make the aws cognito-idp command work for admin and regular users of AppSync (replace the Resource arn with the arn of the user pool[s] you want to control):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cognito-idp:AdminInitiateAuth",
                "cognito-idp:AdminGetUser"
            ],
            "Resource": "arn:aws:cognito-idp:us-east-1:XXXXXXXXXXXXX:userpool/us-east-1_XXXXXXXXX"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cognito-idp:GetUser",
                "cognito-idp:InitiateAuth"
            ],
            "Resource": "*"
        }
    ]
}

Get the Credentials with no App Client Secret

This example is if you did not set the App Client Secret.

You should now be able to get the JWT credentials from the AWS CLI.

This assumes you have set up your ~/.aws/credentials file or whatever is appropriate for your command line environment so that you have the permissions to access this service.

When using the ADMIN_USER_PASSWORD_AUTH

aws cognito-idp admin-initiate-auth --user-pool-id us-east-1_XXXXXXXXXX --auth-flow ADMIN_USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=username1,PASSWORD=XXXXXXXXXXXXX > creds.json

When using the USER_PASSWORD_AUTH

aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=username2,PASSWORD=XXXXXXXXXXXX > creds.json

Of course replace the XXXX's with the actual values.

user-pool-id - The pool id found at the top of the User Pool Client Apps page
client-id - The client-id of the app client you are using
USERNAME - The Username normally used to login to your Amplify app
PASSWORD - The Password normally used to login to your Amplify app

The results will be in creds.json. (You could not use the > creds.json if you want to just see the results)

Get the Credentials when there is an App Client Secret

This assumes you have an App Client that has an app secret key set.

The main thing here is you need to generate a secret hash to send along with the command.

You can do that by creating a little python program to generate it for you when you need it:

#!/usr/bin/env python3

import sys
import hmac, hashlib, base64

if (len(sys.argv) == 4):
    username = sys.argv[1]
    app_client_id = sys.argv[2]
    key = sys.argv[3]
    message = bytes(sys.argv[1]+sys.argv[2],'utf-8')
    key = bytes(sys.argv[3],'utf-8')
    secret_hash = base64.b64encode(hmac.new(key, message, digestmod=hashlib.sha256).digest()).decode()

    print("SECRET HASH:",secret_hash)
else:
    (print("len sys.argv: ", len(sys.argv)))
    print("usage: ",  sys.argv[0], " <username> <app_client_id> <app_client_secret>")

Save the file someplace that you can execute it from like ~/bin/app-client-secret-hash and make it executable (chmod a+x ~/bin/app-client-secret-hash).

You will need:

app-client-id - The client-id of the app client you are using
app-client-secret - The secret of the app client you are using (its on the App Client page of the User Pool)
USERNAME - The Username normally used to login to your Amplify app

To use:

~/bin/app-client-secret-hash  <username> <app_client_id> <app_client_secret>

Where of course you replace the arguments with the actual values.

The result is a secret-hash you will use in the following command to get the actual JWT credentials

aws cognito-idp admin-initiate-auth --user-pool-id us-east-1_XXXXXXXXXX --auth-flow ADMIN_USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=username3,PASSWORD='secret password',SECRET_HASH='secret-hash' > creds.json

You could do the same thing with USER_PASSWORD_AUTH if you nee that instead

aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=rob+admin,PASSWORD=XXXXXXXXX,SECRET_HASH='secret-hash' > creds.json

Using the Credentials

How you use these credentials depends on what tool or how you are trying to access your AppSync APIs.

From some Javascript

You can just add in the IdToken from the creds.json as an Authorization header when you build the request:

function graphQLFetcher(graphQLParams) {
  const APPSYNC_API_URL = "TYPE_YOUR_APPSYNC_URL";
  const credentialsAppSync = {
    Authorization: "eyJraWQiOiI1dVUwMld...",
  };
  return fetch(APPSYNC_API_URL, {
    method: "post",
    headers: {
      Accept: "application/json",
      "Content-Type": "application/json",
      ...credentialsAppSync,
    },
    body: JSON.stringify(graphQLParams),
    credentials: "omit",
  }).then(function (response) {
    return response.json().catch(function () {
      return response.text();
    });
  });
}

If you are using some GraphQL tool that needs to access your AppSync APIs. The tool should have a way that you can supply the token and it will add it as an Authorization header for its own requests.

Do let me know if you have some examples of tools that would make use of this.

References

AWS Amplify and Clojurescript Re-Frame Part 2

Robert J. Berger — Tue, 21 Jan 2020 19:31:08 +0000

Introduction

Now that our Re-Frame app is running, let's deploy it to "production" on AWS Cloudfront using Amplify Console.

The Amplify Console allows you to do a CI style deployment from your git repo with almost no work on your part.

AWS Amplify Console Documentation

Setup Amplify Console

Connect your repo to the Amplify Console

Use your web browser to go to https://console.aws.amazon.com/amplify/home
If you have no previous Amplify Deployments you will see:
- You should select the Get Started under Deploy
Select the appropriate Git service (We used Github)
- It will then go thru the process of allowing you to authenticate that the Amplify service can have the appropriate access to your github repos for the github account/organization you enable
- Will then prompt you for the actual repo and branch you want to build and deployments
- Click NEXT

Note: The previous article Add Serverless AWS Amplify Authentication to a Clojurescript Re-Frame App and this current article are based on branch basic-amplify-with-cloudfront of the omnyway-labs/amplify_cljs_example repo

Configure Basic and Backend build settings

The Backend is the AWS services that were created using the Amplify CLI. For us that was the Authentication service.

The Frontend is our Web App.

You will see the following:
- Change the App name if you want

Backend Deployment Options

You have the choice under Would you like Amplify Console to deploy changes to these resources with your frontend? to:
- Not have it automatically deploy the backend along with your frontend
  - No - only deploy my frontend
- Deploy the same backend for prod as you are using in the dev environment
  - Select dev and not Create new environment
- Deploy a different copy of the backend from your dev environment
  - Select Create new environment
  - Fill in the name of the new environment. In our case prod

Service Role

This is the role that the Amplify Console runs as when setting up your backend services on AWS

If this is the first time thru, you should
- Click Create new role
  - Go thru the pages and accept the pre-selected defaults on each page and create the new service role
- Click the Refresh option
- Use the pulldown to select the role you created
If it wasn't the first time, select the role that already exists

Build Settings

This is the CI Commands to build and deploy your app. It has the default YAML file that at a minimum requires you to set the Frontend build commands and the build output directory.

The Backend commands are fine, but we need to do several things to be able to build our Clojurescript application. You can edit it on the Amplify Console webpage (by clicking on Edit) and / or locally by downloading it and editing locally. In ether case we'll want to download it and put the amplify.yml file in the root directory of our repo.

(Note: Its not clear to me if we really need the file on the console if we have it in our repo. It seems that the one that is in your repo will supersede the one here on the Amplify Console)

Install
- Java
- Leiningen
- shadow-cljs
- Add the following to the frontend prebuild commands:
Set the base directory to where shadow-cljs puts the build assets
Set the frontend build command

frontend:
  phases:
    preBuild:
      commands:
        - amazon-linux-extras enable corretto8
        - yum -y install java-1.8.0-amazon-corretto-devel
        - java -version
        - curl https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein > lein
        - mv lein /usr/local/bin/lein
        - chmod a+x /usr/local/bin/lein
        - lein version
        - npm install -g shadow-cljs
        - npm ci
    build:
      commands: [lein prod]
  artifacts:
    baseDirectory: resources/public/

If all went well, you will eventually have a successful build / deploy
- Click on the Domain link to open the application
The full features of the Authentication Lifecycle are now available
- Click on Create account to create an account
  - The email needs to be a real email address you will receive as confirmation codes will be sent there
- Sign-in then with the credentials you had just created
- And then you are in the app with the option to logout

Set up Custom Domain

If you use AWS Route53 for your DNS
management, the Amplify console can set up a custom DNS Domain and SSL
Certificate to go with your new app!

If you don't use Route53, there is a way to insert a custom DNS but will not show how in this doc.

Select the Domain Management option for the application you just created in the Amplify Console
- Click on Add Domain
Enter the root domain of the route53 zone you want to add this domain to
- We are using a domain zone (lab.omnyway.net) that is a subdomain
  - The important point is it should be the domain that is a zone in your route53
  - The Route53 zone probably has to be the same AWS account as the App
- Click Configure domain
If this is the first time you use this feature, it has to first try some things to get authenticated.
- It will add a weird CNAME to the zone automatically. You don't have to do anything
You could use the defaults if you want
- The root domain to resolve to your webapp
- www.yourdomain to resolve to your webapp
But we have other things in this domain so we want it to just have a single domainname not www
- Click Exclude root so it turns to Include root and dims the pulldown
- Replace www with the domain hostname you want.
  - In this case the name of the app amplify_cljs_example
- Click Save
Assuming it all works
- It will add the CNAME to your route53 zone
- It will create an SSL Certificate and wire it all up
- Will get the domain fully activated
- You can then use the new url shown to access the webapp

Wrap Up

You now have an example of how to create and deploy a Clojurescript Re-Frame SPA with really nice Authentication capabilities with almost no coding.

The next article (Not yet written) will show how to use the AWS Amplify Authentication services to allow access to other standard AWS services.

AWS Amplify and Clojurescript Re-Frame Part 1

Robert J. Berger — Tue, 21 Jan 2020 19:30:35 +0000

Introduction

AWS Amplify

AWS Amplify makes it super simple to build Web and Mobile Apps that use the entire suite of AWS Services with minimal coding by the app developer. Clojurescript and Re-Frame make creating SPA apps a joy. The combination of all three is truly a superpower!

The AWS Amplify CLI makes it easy to setup various AWS services and make them available via the Amplify SDK to your Web or Mobile App. It also has great support for React based apps.

The goal of this project is to show how to be able to use AWS Amplify with Clojurescript / Re-Frame Web Apps.

The initial AWS feature to incorporate is the Amplify Authentication service which has AWS Cognito at its core. Amplify offers a React Higher Order Component that allows you to wrap your JS app with Authentication.

Ensures that your App can only be accessed when authenticated
Handles all the UI and API integration with Cognito
- You do not have to write any of the code
- Sign-up
  - Verify phone_number or email address
- Sign-in
- Sign-out
- Forgot Password
- Password reset
Optionally (and not described in this first article but easy to add)
- Customize Authenticator UI
- Use your own UI components
- AWS Auth for AWS services which require signing requests.
- Social Provider Federation / OAuth
- Force new password
- Multi-Factor Authentication
- User Attributes
- Lambda Triggers

Note: This article and its followup AWS Amplify and Clojurescript Re-Frame Part 2: Deploy to Production Cloudfront are based on branch basic-amplify-with-cloudfront of the omnyway-labs/amplify_cljs_example repo.

Re-Frame / Clojurescript / Shadow-cljs

The main goal of this project is to show how to leverage Amplify in a Clojurescript Single Page App (SPA). In particular, using the re-frame Clojurescript Library.

Re-frame is a functional framework for building SPAs leveraging Reagent. Reagent is a minimalistic interface between ClojureScript and React.

Clojurescript is hosted on Javascript and can naturally interoperate with Javascript in general. This project uses shadow-cljs as the buildtool. Its major superpower is to make it pretty easy to leverage node.js packages. The main features it offers include:

Good configuration defaults so you don't have to sweat the details
Seamless npm integration
Fast builds, reliable caching, ...
Supporting various targets :browser, :node-script, :npm-module, :react-native, :chrome-extension, ...
Live Reload (CLJS + CSS)
CLJS REPL
Code splitting (via :modules)

Put together, these tools allows one to build SPAs in a functional / Clojure centric way, but still leverage all the wonders of React and node.

That being said. There are a lot of learning curves that also need to be climbed to make it all work together.

This project is an attempt to make it easier for folks to jump right in. It can be used as a starting point to build your own SPA with sophisticated Authentication services.

This article and the associated github repo omnyway-labs/amplify_cljs_example should tell you everything you need from scratch to get you to a working basis of a re-frame SPA running on your local machine with full AWS Cognito Simple Authentication backend. The amount of code you have to write to make it work is surprisingly trivial due to the scaffolding features of re-frame and the AWS generation / auto-configuration of Amplify.

The second article in this series Deploy to CloudFront with Amplify Console shows how to deploy the SPA to AWS CloudFront with minimal effort.

Assumptions

You have already installed the following on your laptop / dev machine:

An AWS Account that you have administrative access to
git (Ideally have a github account)
Java
Node.js and npm
Clojure
Clojurscript
shadow-cljs
leiningen
Have an editor and know how to use it

Initial Setup

We're using the re-frame scaffold template to create the initial app. We're not going to add much to the app in this article other than the Amplify Authentication wrapper.

Later articles will add more functionality so we're including some extras like garden for doing CSS in clojure. More importantly we're including re-com Which is a library of ClojureScript UI components, built on top of Reagent. It integrates with re-frame in a very functional / clojure way.

10x is a very nice debugging dashboard for re-frame. Again, won't be using it until later articles but its something you will want to have if you use this for building on top of.

Generate the scaffold

lein new re-frame amplify_cljs_example +10x +cider +re-com +test +garden
cd amplify_cljs_example
lein garden once
git init

Update .gitignore by adding:

/resources/public/css/screen.css
.shadow-cljs
node_modules

More setup; build and run the local instance of the initial scaffolded SPA

git add -A
git commit -a -m "Initial commit after lein new re-frame amplify_cljs_example +10x +cider +re-com +test +garden"
npm install
git add package-lock.json
git commit -m "package-lock for npm" package-lock.json
lein garden once
lein dev

The lein dev will build and run the application locally. It doesn't exit, it keeps running, monitoring the app files. If there are any changes it will automatically hot load the application. Lein is effectively running shadow-cljs watch app

The lein dev probably added more packages for npm and so you should update your git in another shell window in the same directory

git commit -m "lein updates to package.json" package-lock.json  package.json

Ensure the basics are working

So if the lein dev is running properly, it will end with:

[:app] Build completed. (1659 files, 1658 compiled, 0 warnings, 46.89s)

At that point, you should be able to:

Go to http://localhost:8280 in a browser (Chrome is optimal)
Should look something like:

Set up AWS Amplify

You should be in the top level of the amplify_cljs_example directory
Assuming you don’t already have AWS Amplify installed on your dev machine:

npm install -g @aws-amplify/cli
amplify configure

Follow the instructions as described in Installing & Configuring the AWS Amplify CLI - YouTube to configure you local dev amplify environment to manage your AWS environment
Initialize the Amplify project.
- Mostly accept the defaults or as appropriate for you
- Start command : lein dev
- AWS Profile yes and use the one you created when you did amplify configure which by default is default

> amplify init
Scanning for plugins...
Plugin scan successful
Note: It is recommended to run this command from the root of your app directory
? Enter a name for the project reframerecomamplifye
? Enter a name for the environment dev
? Choose your default editor: Emacs (via Terminal, Mac OS only)
? Choose the type of app that you're building javascript
Please tell us about your project
? What javascript framework are you using react
? Source Directory Path:  src
? Distribution Directory Path: build
? Build Command:  lein prod
? Start Command: lein dev
Using default provider  awscloudformation

For more information on AWS Profiles, see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html

? Do you want to use an AWS profile? Yes
? Please choose the profile you want to use default
⠧ Initializing project in the cloud...

CREATE_IN_PROGRESS AuthRole                                AWS::IAM::Role             Fri Oct 04 2019 22:50:40 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS DeploymentBucket                        AWS::S3::Bucket            Fri Oct 04 2019 22:50:40 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS UnauthRole                              AWS::IAM::Role             Fri Oct 04 2019 22:50:40 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS AuthRole                                AWS::IAM::Role             Fri Oct 04 2019 22:50:39 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS DeploymentBucket                        AWS::S3::Bucket            Fri Oct 04 2019 22:50:39 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UnauthRole                              AWS::IAM::Role             Fri Oct 04 2019 22:50:39 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS reframerecomamplifye-dev-20191004225035 AWS::CloudFormation::Stack Fri Oct 04 2019 22:50:36 GMT-0700 (Pacific Daylight Time) User Initiated
⠴ Initializing project in the cloud...

CREATE_COMPLETE UnauthRole AWS::IAM::Role Fri Oct 04 2019 22:50:52 GMT-0700 (Pacific Daylight Time)
⠏ Initializing project in the cloud...

CREATE_COMPLETE AuthRole AWS::IAM::Role Fri Oct 04 2019 22:50:52 GMT-0700 (Pacific Daylight Time)
⠹ Initializing project in the cloud...

CREATE_COMPLETE DeploymentBucket AWS::S3::Bucket Fri Oct 04 2019 22:51:01 GMT-0700 (Pacific Daylight Time)
⠇ Initializing project in the cloud...

CREATE_COMPLETE reframerecomamplifye-dev-20191004225035 AWS::CloudFormation::Stack Fri Oct 04 2019 22:51:03 GMT-0700 (Pacific Daylight Time)
✔ Successfully created initial AWS cloud resources for deployments.
✔ Initialized provider successfully.
Initialized your environment successfully.

Your project has been successfully initialized and connected to the cloud!

Add the Amplify Authentication service
- Details at AWS Amplify Authentication
- Sign-in: Selecting 'Email' and/or 'Phone Number' will allow end users to sign-up using these values. Selecting 'Username' will require a unique username for users.

> amplify add auth
Using service: Cognito, provided by: awscloudformation

 The current configured provider is Amazon Cognito.

 Do you want to use the default authentication and security configuration? Default configuration
 Warning: you will not be able to edit these selections.
 How do you want users to be able to sign in? Email
 Do you want to configure advanced settings? No, I am done.
Successfully added resource reframerecomamplifye16cd456a locally

Push the auth feature / config to AWS

> amplify auth push

Current Environment: dev

| Category | Resource name                | Operation | Provider plugin   |
| -------- | ---------------------------- | --------- | ----------------- |
| Auth     | reframerecomamplifye16cd456a | Create    | awscloudformation |
? Are you sure you want to continue? Yes
⠹ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS UpdateRolesWithIDPFunctionRole          AWS::IAM::Role             Fri Oct 04 2019 23:08:01 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS authreframerecomamplifye16cd456a        AWS::CloudFormation::Stack Fri Oct 04 2019 23:08:01 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS authreframerecomamplifye16cd456a        AWS::CloudFormation::Stack Fri Oct 04 2019 23:08:01 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UpdateRolesWithIDPFunctionRole          AWS::IAM::Role             Fri Oct 04 2019 23:08:00 GMT-0700 (Pacific Daylight Time)
UPDATE_IN_PROGRESS reframerecomamplifye-dev-20191004225035 AWS::CloudFormation::Stack Fri Oct 04 2019 23:07:57 GMT-0700 (Pacific Daylight Time) User Initiated
⠇ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS reframerecomamplifye-dev-20191004225035-authreframerecomamplifye16cd456a-19BSJYQRK36 AWS::CloudFormation::Stack Fri Oct 04 2019 23:08:01 GMT-0700 (Pacific Daylight Time) User Initiated
⠋ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS SNSRole AWS::IAM::Role Fri Oct 04 2019 23:08:08 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS SNSRole AWS::IAM::Role Fri Oct 04 2019 23:08:07 GMT-0700 (Pacific Daylight Time)
⠴ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE UpdateRolesWithIDPFunctionRole AWS::IAM::Role Fri Oct 04 2019 23:08:15 GMT-0700 (Pacific Daylight Time)
⠋ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE SNSRole AWS::IAM::Role Fri Oct 04 2019 23:08:20 GMT-0700 (Pacific Daylight Time)
⠙ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE    UserPool AWS::Cognito::UserPool Fri Oct 04 2019 23:08:27 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UserPool AWS::Cognito::UserPool Fri Oct 04 2019 23:08:27 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS UserPool AWS::Cognito::UserPool Fri Oct 04 2019 23:08:24 GMT-0700 (Pacific Daylight Time)
⠇ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE    UserPoolClientWeb AWS::Cognito::UserPoolClient Fri Oct 04 2019 23:08:33 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UserPoolClientWeb AWS::Cognito::UserPoolClient Fri Oct 04 2019 23:08:33 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_COMPLETE    UserPoolClient    AWS::Cognito::UserPoolClient Fri Oct 04 2019 23:08:33 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UserPoolClient    AWS::Cognito::UserPoolClient Fri Oct 04 2019 23:08:32 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS UserPoolClientWeb AWS::Cognito::UserPoolClient Fri Oct 04 2019 23:08:31 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UserPoolClient    AWS::Cognito::UserPoolClient Fri Oct 04 2019 23:08:31 GMT-0700 (Pacific Daylight Time)
⠙ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS UserPoolClientRole AWS::IAM::Role Fri Oct 04 2019 23:08:36 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS UserPoolClientRole AWS::IAM::Role Fri Oct 04 2019 23:08:36 GMT-0700 (Pacific Daylight Time)
⠏ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE UserPoolClientRole AWS::IAM::Role Fri Oct 04 2019 23:08:48 GMT-0700 (Pacific Daylight Time)
⠹ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE    UserPoolClientLambda AWS::Lambda::Function Fri Oct 04 2019 23:08:52 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UserPoolClientLambda AWS::Lambda::Function Fri Oct 04 2019 23:08:52 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS UserPoolClientLambda AWS::Lambda::Function Fri Oct 04 2019 23:08:52 GMT-0700 (Pacific Daylight Time)
⠸ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS UserPoolClientLambdaPolicy AWS::IAM::Policy Fri Oct 04 2019 23:08:57 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS UserPoolClientLambdaPolicy AWS::IAM::Policy Fri Oct 04 2019 23:08:56 GMT-0700 (Pacific Daylight Time)
⠼ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE UserPoolClientLambdaPolicy AWS::IAM::Policy Fri Oct 04 2019 23:09:02 GMT-0700 (Pacific Daylight Time)
⠋ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS UserPoolClientLogPolicy AWS::IAM::Policy Fri Oct 04 2019 23:09:06 GMT-0700 (Pacific Daylight Time)
⠸ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS UserPoolClientLogPolicy AWS::IAM::Policy Fri Oct 04 2019 23:09:06 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
⠸ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE UserPoolClientLogPolicy AWS::IAM::Policy Fri Oct 04 2019 23:09:12 GMT-0700 (Pacific Daylight Time)
⠸ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS UserPoolClientInputs Custom::LambdaCallout Fri Oct 04 2019 23:09:15 GMT-0700 (Pacific Daylight Time)
⠹ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS IdentityPool         AWS::Cognito::IdentityPool Fri Oct 04 2019 23:09:23 GMT-0700 (Pacific Daylight Time)
CREATE_COMPLETE    UserPoolClientInputs Custom::LambdaCallout      Fri Oct 04 2019 23:09:19 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UserPoolClientInputs Custom::LambdaCallout      Fri Oct 04 2019 23:09:19 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
⠼ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE    IdentityPool AWS::Cognito::IdentityPool Fri Oct 04 2019 23:09:25 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS IdentityPool AWS::Cognito::IdentityPool Fri Oct 04 2019 23:09:24 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
⠋ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS IdentityPoolRoleMap AWS::Cognito::IdentityPoolRoleAttachment Fri Oct 04 2019 23:09:28 GMT-0700 (Pacific Daylight Time)
⠴ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE    reframerecomamplifye-dev-20191004225035-authreframerecomamplifye16cd456a-19BSJYQRK36 AWS::CloudFormation::Stack               Fri Oct 04 2019 23:09:32 GMT-0700 (Pacific Daylight Time)
CREATE_COMPLETE    IdentityPoolRoleMap                                                                  AWS::Cognito::IdentityPoolRoleAttachment Fri Oct 04 2019 23:09:30 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS IdentityPoolRoleMap                                                                  AWS::Cognito::IdentityPoolRoleAttachment Fri Oct 04 2019 23:09:30 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
⠧ Updating resources in the cloud. This may take a few minutes...

CREATE_COMPLETE authreframerecomamplifye16cd456a AWS::CloudFormation::Stack Fri Oct 04 2019 23:09:36 GMT-0700 (Pacific Daylight Time)
⠏ Updating resources in the cloud. This may take a few minutes...

CREATE_IN_PROGRESS UpdateRolesWithIDPFunctionOutputs Custom::LambdaCallout Fri Oct 04 2019 23:09:41 GMT-0700 (Pacific Daylight Time)
CREATE_COMPLETE    UpdateRolesWithIDPFunction        AWS::Lambda::Function Fri Oct 04 2019 23:09:39 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS UpdateRolesWithIDPFunction        AWS::Lambda::Function Fri Oct 04 2019 23:09:39 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
CREATE_IN_PROGRESS UpdateRolesWithIDPFunction        AWS::Lambda::Function Fri Oct 04 2019 23:09:38 GMT-0700 (Pacific Daylight Time)
⠹ Updating resources in the cloud. This may take a few minutes...

UPDATE_COMPLETE                     reframerecomamplifye-dev-20191004225035 AWS::CloudFormation::Stack Fri Oct 04 2019 23:09:47 GMT-0700 (Pacific Daylight Time)
UPDATE_COMPLETE_CLEANUP_IN_PROGRESS reframerecomamplifye-dev-20191004225035 AWS::CloudFormation::Stack Fri Oct 04 2019 23:09:47 GMT-0700 (Pacific Daylight Time)
CREATE_COMPLETE                     UpdateRolesWithIDPFunctionOutputs       Custom::LambdaCallout      Fri Oct 04 2019 23:09:45 GMT-0700 (Pacific Daylight Time)
CREATE_IN_PROGRESS                  UpdateRolesWithIDPFunctionOutputs       Custom::LambdaCallout      Fri Oct 04 2019 23:09:44 GMT-0700 (Pacific Daylight Time) Resource creation Initiated
✔ All resources are updated in the cloud

Install the Amplify Node modules we’ll be using for Authentication

npm add aws-amplify aws-amplify-react

Copy the css for the amplify UI widgets into the proper resource for our app’s css to be picked up
- Note: I was looking for a better way to do this that is more via configuration. If anyone knows a better way please let me know

cp node_modules/@aws-amplify/ui/dist/style.css resources/public/css/aws-amplify-ui-style.css

Update resources/public/index.html to incorporate that css by adding the following line in the <head> section

<link href="css/aws-amplify-ui-style.css" rel="stylesheet" type="text/css">

Get the changes into git

git add amplify resources/public/css/aws-amplify-ui-style.css
git commit -m "Added and configured Amplify" .gitignore package-lock.json package.json amplify resources/public/index.html resources/public/css

Your application at http://localhost:8280 should still be running fine assuming you still have the lein dev process running

Update the code to enable the Authenticator

Edit `src/cljs/re_frame_re_com_amplify_exp/core.cljs`

Add the following lines to the require block. This is what will import the node js into your Clojurescript

   ["aws-amplify" :default Amplify :as amp]
   ["aws-amplify-react" :refer (withAuthenticator)]
   ["/aws-exports.js" :default aws-exports]

Add the following def after dev-setup function

This is where you are wrapping the Amplify HOC around your application. The withAuthenticator is the HOC function being pulled in from the Amplify Auth SDK. It ensures that your app only is accessed after authentication and it handles all the Frontend UI and communication with the Backend to implement the full suite of Authentication.

(def root-view
  (reagent/adapt-react-class
   (withAuthenticator
    (reagent/reactify-component views/main-panel) true)))

Change the function mount-root to be:

(defn ^:dev/after-load mount-root []
  (re-frame/clear-subscription-cache!)
  (.configure Amplify aws-exports)
  (re-frame/dispatch-sync [::events/initialize-db])
  (reagent/render [root-view]
                  (.getElementById js/document "app")))

Remove the following line from init function:

This makes it so that the re-frame db is inited after the cache is cleared. As far as I can tell the Scaffolding puts it in the wrong place

(re-frame/dispatch-sync [::events/initialize-db])

The final file should be:

(ns amplify_cljs_example.core
  (:require
   [reagent.core :as reagent]
   [re-frame.core :as re-frame]
   [amplify_cljs_example.events :as events]
   [amplify_cljs_example.views :as views]
   [amplify_cljs_example.config :as config]
   ["aws-amplify" :default Amplify :as amp]
   ["aws-amplify-react" :refer (withAuthenticator)]
   ["/aws-exports.js" :default aws-exports]
   ))

(defn dev-setup []
  (when config/debug?
    (println "dev mode")))

(def root-view
  (reagent/adapt-react-class
   (withAuthenticator
    (reagent/reactify-component views/main-panel) true)))

(defn ^:dev/after-load mount-root []
  (re-frame/clear-subscription-cache!)
  (.configure Amplify aws-exports)
  (re-frame/dispatch-sync [::events/initialize-db])
  (reagent/render [root-view]
                  (.getElementById js/document "app")))

(defn init []
  (dev-setup)
  (mount-root))

The display should look like this:

You should click the button on the debugger to have it be in its own window as it will block the “sign-out” button later
Commit the changes to git

git commit -m "App updated to require Cognito Authenticator before any other operation" src/cljs/re_frame_re_com_amplify_exp/core.cljs

Wrap Up

You now have a template of how to create a basic Re-Frame SPA with authentication from AWS Amplify with almost not coding and without having to set up your own backend authentication infrastructure. You can continue with this to use other AWS Amplify authentication features (Social Provider Federation, Multi-factor auth, etc) as well as the other Amplify services such as graphql access to DynamoDB, Push notifications, etc.).

The next article, AWS Amplify and Clojurescript Re-Frame Part 2: Deploy to Production Cloudfront, will show how to deploy it to "production" on AWS Cloudfront using Amplify Console so that the Re-Frame SPA can be used on the general Internet also without you having to set up any servers yourself.

DEV Community: Robert J. Berger

Building AWS Ruby Lambdas that Require Gems with Native Extension

What's the Problem?

Bundle in a Quick Lambda Linux Docker Container

Dissecting the Commands

Zip it Up Good

Upload to Lambda function

Wrapping Things Up

About the author

Python src Layout for AWS Lambdas

Example src layout for an AWS Lambda

Example Poetry pyproject.toml

Set the Lambda Handler to be the "two dot" format

Example src/my_lambda/handler.py

Building and deploying the Lambda

Building the distribution

Create the Lambda function

If you already have created the Lambda function

Test the Lambda

Running tests with Pytest

References on src layout

Deploy EventCatalog to AWS CloudFront with Google SSO Access Control via Terraform

Table of Contents

Create EventCatalog Project

Requirements

Generate the Scaffold Project Website

Create the Terraform to deploy to Cloudfront

Create the file main.tf

Create the lambda.tf file

Create the cloudfront.tf file

Create the variables.tf file

Create the sandbox.tfvars file

Create a placeholder lambda code zip file

Initial deployment with temp lambda@edge code

Build the Lambda@edge code with Widen/cloudfront-auth

Create the OAuth Credentials in the Google developers console

Create a new Project

Create OAuth Consent and Credentials

Generate the code for Lambda@edge

Deploy the EventCatalog content to S3

Manual deployment

Deployment with CircleCi

Deploy the new lambda@edge code with terraform

Improvements? Suggestions? Alternatives?

About the Author

Connect Quicksight to RDS in a private VPC

Quicksight VPC Connection Requirements

Subnet Info

Get VPC Info

Pick a Subnet for the Quicksight Network Interface to Use

Find the subnets used on the target DB

Security Group

Inbound Rules

Outbound Rules

DNS Resolver Endpoints (optional)

Route53 Resolver Inbound Endpoints

Create a Security Group for the Resolver

Inbound Rules

Outbound Rules

Setup the Route53 Endpoint Resolver

Update the Quicksight Security Group for DNS

Create the Actual VPC Connection

Create a Dataset using the VPC Connection

Fill in the New Aurora data source form

Click on Validate Connection

If the VPC Connection fails to Validate

Check for basic connectivity

Find the ID of the VPC Connection Network Interface

Find the ID of the Target DB Network Interface

Run the Reachability Analyzer

Conclusion

How to Use Amplify Studio Figma Connector with Clojurescript

Amplify Studio / Figma / Clojurescript / Reagent Tutorial

Tooling Used in the project:

Prerequisites

Setup an Amplify Studio Project

Creating an Amplify Studio App with Clojurescript

Create a git repo with shadow-cljs / reagent scaffolding

Add webpack and related dependencies

Update your local git repo

Running tests with `Pytest`

References on `src layout`

Create the file `main.tf`

Create the `lambda.tf` file

Create the `cloudfront.tf` file

Create the `variables.tf` file

Create the `sandbox.tfvars` file

Quicksight `VPC Connection` Requirements

Fill in the `New Aurora data source` form

Click on `Validate Connection`

Move `public/index.html` to `public/index.html.tmpl`

Edit `public/index.html.tmpl`

Add the dependencies for the `require`

Update the `app` function

Update the `main` function

The full `webpack.config.js`