DEV Community: Mirko Vukušić

VimWiki: how to automate wikis per project folder (Neovim)

Mirko Vukušić — Fri, 11 Feb 2022 00:06:21 +0000

Neovim is grabbing more and more pieces of my workflow from other systems, and last two pieces are notes and tasks.
Org-mode plugins are great, but focused more on tasks.

However, one small thing almost made me adopt org-mode instead of VimWiki. That is ease of creating .org files anywhere, and that usually means in many of my project folders. I like to have all my project related stuff inside individual project folders, leave it or hide it from source control, backup, sync, ... I don't like scattered files.

Of course, VimWiki can have multiple wikis... just add them to your config. Ah, not my preferred way of managing any list. Who wants to edit wiki config manually every time there is a new project.

So this is what I did to automate it. Please note it requires Neovim (script is written in Lua), Linux (or better to say find which I use for searching folders, and optionally Fzf (but this can be easily changed).

General idea

... is to use find to search for a wiki folder recursively in projects_folder. Then, result list is used to dynamically populate VimWiki global variable containing list of wikis (g:vimwiki_list) and tell VimWiki to refresh the list. Now, VimWiki's <Leader>ws is populated with project wikis! Wikis defined in config are there too, so you can still manually add wikis outside of projects folder if you want.

Using VimWiki's original wiki search feature

Minimal code for the above is:

-- configuration
local config = {
    projectsFolder = '/home/your_user/work/', --full path without ~
    maxDepth = 3,
    ignoreFolders = { 'node_modules', '.git' },
    rootWikiFolder = '_wiki',
    wikiConfig = { syntax='markdown', ext='.md' }
}

-- store original vimwiki_list config, we will need it later
-- !!!make sure vimwiki plugin is loaded before running this!!!
if vim.g.vimwiki_list == nil then
    error('VimWiki not loaded yet! make sure VimWiki is initialized before my-projects-wiki')
else
    _G.vimwiki_list_orig = vim.fn.copy(vim.g.vimwiki_list) or {}
end

-- function to update g:vimwiki_list config item from list of subfolder names (append project wikis)
--   this way, orginal <Leader>ws will get new project wikis in the list and also keep ones from config
local function updateVimwikiList(folders)
    local new_list = vim.fn.copy(vimwiki_list_orig)
    for _, f in ipairs(folders) do
        local item = {
            path = config.projectsFolder..f,
            syntax = config.wikiConfig.syntax,
            ext = config.wikiConfig.ext
        }
        table.insert(new_list, item)
    end
    vim.g.vimwiki_list = new_list
    vim.api.nvim_call_function('vimwiki#vars#init',{})
end

-- function to search project folders for root wiki folders (returns system list)
local function searchForWikis()
    local command = 'find ' .. config.projectsFolder ..
        ' -maxdepth ' .. config.maxDepth
    if #config.ignoreFolders > 0 then command = command .. " \\(" end
    for _, f in ipairs(config.ignoreFolders) do
        command = command .. " -path '*/"..f.."/*' -prune"
        if next(config.ignoreFolders,_) == nil then
            command = command .. " \\) -o"
        else
            command = command .. " -o"
        end
    end
    command = command .. ' -type d -name ' .. config.rootWikiFolder
    command = command .. ' -print | '
    command = command .. ' sed s#' .. config.projectsFolder .. '##'
    local list = vim.api.nvim_call_function('systemlist', {command})
    return list
end

Now, you just have to call updateVimwikiList(searchForWikis()) somewhere to populate VimWiki with project wikis. There are so many options to do this. You can define a new command to do it manually, you can remap VimWiki's <Leader>ws to execute it before displaying a list of wikis, or you can simply call it at the bottom of the script to refresh the list on Neovim start (downside of this method is that new _wiki folders will not be detected till you restart).

Extend it with Fzf fuzzy-find menu

However, I took a different approach. Script is really fast, so I don't mind running it every time I'm searching for wikis. Also, I use Fzf fuzzy-finder. So I decided to feed the list to Fzf for a nice fuzzy-find menu. This will replace default VimWiki's wiki select feature which is not very nice, especially for larger number of projects.

Here is the additional code, just below the above code:

-- wrapper for Vimwiki's goto_index() to bypass :VimWikiUISelect and use FZF instead
-- if wiki is passed to the function, index page is opened directly, bypassing FZF
function _G.ProjectWikiOpen(name)
    -- show fzf wiki search if no wiki passed
    if not name then
        local wikis = searchForWikis()
        updateVimwikiList(searchForWikis())
        for _,f in ipairs(vimwiki_list_orig) do table.insert(wikis,f.path) end
        local options = {
            sink = function(selected) ProjectWikiOpen(selected) end,
            source = wikis,
            options = '--ansi --reverse --no-preview',
            window = {
                width = 0.3,
                height = 0.6,
                border = 'sharp'
            }
        }
        vim.fn.call('fzf#run', {options})
    else
        for i, v in ipairs(vim.g.vimwiki_list) do
            if v.path == name or v.path == config.projectsFolder..name then
                vim.fn.call('vimwiki#base#goto_index',{i})
                return
            end
        end
        print("Error. Selected project wiki not found")
    end
end


-- add commands
vim.api.nvim_command([[command! -nargs=? ProjectWikiOpen lua ProjectWikiOpen(<f-args>)]])

-- add keybindings
vim.api.nvim_set_keymap("n", "<Leader>wp", ":ProjectWikiOpen<CR>", { noremap=true })

So, a new :ProjectWikiOpen command is registered and mapped to wp. I like this key combo better than original <Leader>ws. This leaves you with a very simple workflow... just add _wiki folder to any project and hit <Leader>wp. Fzf menu will open and it will already be there. Just select it and VimWiki takes over.

todo

1) Finish Project/Session switcher script - For me, everything is a project and everything project related is in a project folder. That's why I also have a few scripts to easily manage/switch my projects in Neovim. Those integrate project sessions (using mksession), automatic fuzzy-finder for projects (using find to search for .git folders to find roots), and Kitty terminal project launcher integrated in Dmenu which starts project environment using Kitty terminal: opens few Kitty windows and tabs with Neovim (session preloaded), LazyGit, few terminals, etc. It's almost done so hopefully I find the time to publish that too.

2) Finish synchronization with mobile phone - I want my wikis on the go too. This is almost done with Syncthing client on my laptop, server (Nextcloud) and mobile phone. Wikis are pushed and synchronized on all devices. Currently trying Obsidian Android app to read/manage wikis on the mobile side.

3) Integrate VimWiki tasks and taskwarrior (or similar)

How to toggle Polybar on Spectrwm tiling wm

Mirko Vukušić — Wed, 09 Feb 2022 02:42:45 +0000

Just a short one on this problem with combination of Spectrwm tiling window manager and Polybar. Managed to sort it out after hours without finding a solution on the web. So here is my hack.

First, the problem... Once you replace the bar that comes with Spectrwm, with Polybar, you have to reserve a portion on top of the screen for Polybar. That's OK, but toggling Polybar leaves that reserved space empty so again you don't get a full screen. On the other hand, if you don't reserve that space, Polybar is displayed above your windows which is also not nice.

My solution is simple:

Leave original bar ON with minimal settings, and remove reserved space for Polybar if you had it configured before (region in config). Only Spectrwm bar config items you need are:

bar_enabled     = 1
bar_border_width    = 0
bar_font        = <any font, just adjust size so bar is as large as Polybar>
bar_format      = ' # <--- something minimal so bar is almost not visible

make sure your original keybinding to toggle Spectrwm bar is still active (Super+b)
install xdotool. Miniature program that can send fake keys to Xorg
write shell script, I named it fullscreen-toggle.sh and put it somewhere (mine are in ~/scripts/). Also make it executable:

#!/bin/sh
xdotool key super+b
polybar-msg cmd toggle

Script is really simple. First, xdotool toggles Spectrwm bar by sending fake keypresses. Then we use polybar-msg to send remote command to Polybar to toggle.

Add new keybinding to Spectrwm conf (~/.spectrwm.comf). I used MOD+Shift+f:

program[toggle_bars] = ~/scripts/fullscreen-toggle.sh
bind[toggle_bars] = MOD+Shift+f

... and we're done! Restart Spectrwm (Super+q) and try it out. Make sure your keybinding was free before assigning it. Also, try running shell script from terminal for test.

Am I the only one frustrated with oh-my-zsh?

Mirko Vukušić — Tue, 11 May 2021 11:19:45 +0000

I feel lonely :) Help me stop this feeling I'm the only one. Just switched from bash to zsh, and while I completely get why somebody would want to use oh-my-zsh, and I don't have anything against the project, it just seems like everybody is using it!? It's difficult to Google for any zsh configuration help/tutorials without ending up with oh-my-zsh. Again, it's not that oh-my-zsh itself bothers me, it bothers me that all my search results are infested with it :)

Changed my Linux "stack" after 10 years

Mirko Vukušić — Sun, 04 Oct 2020 14:58:59 +0000

After 10 years and a simple post on dev.to I completely changed my Linux distro and tools.

I wonder if you did something like this and what was your change? I want to hear long time Linux users, not a fresh ones. I'm interested about a change, not tools themselves.

My change was:

Mint/Cinnamon to Arch/Xmonad (after 10 years)
WebStorm to Vim (after 2 years, was Sublime before)
MC to Vifm (after probably 30 years and Norton Commander in DOS)

... Or in general: Mouse to keyboard :)

Do not update Linux Mint 19 !!! Bricked systems reported

Mirko Vukušić — Tue, 29 Sep 2020 20:49:04 +0000

Well, it seems Linux Mint 19 update from today bricks a lot of systems! My friend's one included. More info on their thread https://forums.linuxmint.com/viewtopic.php?f=46&t=331520

Uncle Bob's clean code lecture

Mirko Vukušić — Fri, 07 Aug 2020 22:22:37 +0000

https://youtu.be/7EmboKQH8lM

Just had to reshare this. Amazing lecture by Uncle Bob, so mush wider than a title says.

How to migrate AWS Cognito User Pool and DynamoDB Table (script)

Mirko Vukušić — Sun, 12 Jul 2020 10:59:27 +0000

Yesterday I had to migrate my AWS app to a new setup on CloudFormation. Code is easy but Cognito User Pool and DynamoDB table(s) took some time to investigate. Here is my experience and a solution (for small to medium apps).

Migrating Cognito User Pool

Note that there are several ways to migrate Cognito User pool, two main ones are described here: https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-import-users.html

Option 1) Using "seamless" Lambda migration

Basically you keep both pools active; app is normally linked to the new pool but on every failed login Lambda will check user against the old pool and migrate it to a new pool seamlessly (if exists and credentials are ok).

Option 2) Import your users from CSV file

Simple but not seamless. Using Cognito console you can import users from CSV file, but they'll be put into RESET_REQUIRED status. Then, your app needs to be changed a bit to forward users with this status to "forgot password" workflow. There they will re-confirm their email and it's done. Getting a CSV file to import is another story as Cognito console does not export users (yet).

Why I chose option 2 and import CSV?

because it is much simpler for small to medium apps
because I didn't want to keep both user pools active for that long (my users migrate very slowly)
because my database links some data with username (which is a login alias for email) so in "seamless" method there was a chance that a completely new user would "steal" old username (although with new email) exposing that data to him.

Exporting users from Cognito user pool is still not available in console, but a cognito-backup-restore does the job very well. However, it does not (yet) create proper CSV for import through console. Some simple Node scripting and it's done (link to code at the end).
Note that cognito-backup-restore does have a restore feature too, but it sends out emails to users which was not what I wanted. I wanted this migration silent.

Migrating DynamoDB table(s)

Again, there are several options to migrate DynamoDB. Amazon suggests AWS Data Pipeline but for small to medium tables that's an overkill. Many other variations and methods to do this exist online, free or commercial. Also many different JSON formats are used in diffrente tools which require some parsing. At the end I decided to use AWS CLI commands wrapped in a Node script to automate the process.

step 1: aws dynamodb scan downloads DynamoDB JSON
step 2: aws dynamodb put-item in a loop sends all records to a new table, one by one

batch-write was something to think about (it can do 25 records in a batch) but my table was small enough to do it this way.

Complete code is on GitHub: cognito-dynamodb-migrate

Javascript performance benchmarking (aka JSPerf) with JSBench.me

Mirko Vukušić — Sun, 12 Jul 2020 00:24:15 +0000

Disclaimer: this is my app and the post is mostly announcement

https://jsbench.me

Long time ago, JSPerf was down for some time and since I was learning React and JSPerf was server-side app, I decided to build my own, SPA version in React. Fast forward to today... JSBench.me is out of beta. v1.0.0 is released.

What is it for?

Well, if you use(d) JSPerf, then explanation is not needed. If not, shortest description would be that it is Javasscript playground (like JSBin, JSFiddle) but for testing performance of your snippets. In the background Benchmark.js runs all tests and JSBench.me is UI for it.

Question? Do you benchmark your Javascript code?

I got a habit of doing it, more as a learning tool than a development tool. Once you realize one way of doing something is faster, you adopt it for future.

What's under the hood?

As said, it is an SPA, built with React and Mobx. Back office is relatively complex mix of AWS services: Cloud Formation, API Gateway, Cognito, Lambda and DynamoDB. So, complete project is in the cloud depending on AWS services. V1 also uses AWS SAM with Cloud Formation to build complete infrastructure from yaml files. Whole project tought me a lot and will produce several posts here, one of which is already here.

Features and what's new in this version

Other than standard test running, you always could save/publish and link your test suites. Voting is also possible. Last version (other than complete rewrite to latest libraries and Typescript) adds many UI helpers that users requested like: sorting tests (drag&drop or buttons), running individual tests in the suite, dark & light theme, manual&auto resizing of code editors, etc. More in the change log

Search and sort of articles are confusing to me

Mirko Vukušić — Sat, 27 Jun 2020 12:26:32 +0000

Reading dev.to for a few days now. After reading some fresh posts, often I want to check some topics of interest using a search. But my first impression is that it just doesn't feel right. Mostly its about getting old articles in results, so quickly I have nothing more to read on the topic, ot it just appears like it, or Im doing it wrong.

Here is a story of my first impressions...

Day 1)
I'm interested in "aws" topics. I know there's #aws tag for it, let me find it. Menu - tags... Oh, list is huge, lets try to search for a tag... Nope. search above searches the site, not tags.
Ok, lets go alphabetically... Nope, they're not alphabetically sorted. Well, I'm not browsing all those. Let's search for '#aws', yes! Got it. Wait ... Too many articles are 2 years old. It cannot be that. Open first article, click on #aws tag... Aha, completely different thing.

Day 2)
I've read top #aws articles, let's dig deeper. I want to see articles on AWS Sam. Hmm, can I search by tag? Let's search for: #aws sam
Good... But hey... Again top articles are 2 years old. There must be more on dev.to on Sam!? Scrolling down I realize there are newer articles deep down. Uh, don't want that. I've read all articles at the top, how am I going to continue reading tomorrow? I don't have a clue how those articles are sorted. It seems that list in search results and list created by clicking on the #tag are sorted differently, and there's no "sort by date" in any. I don't see the other way but to go through complete list every day, not to miss new ones but continuing to read beyond the point where I stopped yesterday.

There must be something about this on dev.to. Let me search for: #devto sort
Whaaat!? I get list of people in results? Confused. Maybe this is just displayed when there are no search result to display? Let me try searching for: 'gshdyejsjsieheh'...
Oh, now i get "no results match that query". So what's with that people in search results. Still confused

Day 3)
Dev.to just appears like it has less (and older) content than it really has. At least in my use scenario. I mostly use homepage and 'latest' and a few tags I subscribed to. I feel like I'm missing something.

What am I getting\doing wrong?

Build & deploy all AWS resources as easy as: npm run deploy-prod (AWS Sam worflow)

Mirko Vukušić — Tue, 23 Jun 2020 21:07:25 +0000

After manually managing my AWS resources I finally decided to use Sam templates and put everything in Git. Easy deploying to prod and test stages was also a requirement. Here is a short overview of some issues I ran into and how I resolved them. It is by no means a detailed step by step guide, but just a quick overview with some problems I ran into and solutions.

Disclaimer: This is my first experience with AWS Sam. I probably did some things sub-optimal or wrong. Idea is to share and get comments from those with more experience. It may also help those starting with AWS Sam, to speed up the process and save a week I spend investigating.
Also note that current Sam version I'm using is 0.52.0 and I'm on Linux.

sam-cli installation

First requirement is sam-cli. I'm not getting into details of installing and configuring it, but you can find it here.

You will also have to add sam-cli user an permissions on AWS IAM. This can be a slow process of trial and error. Depending on your resources, and motivation to close down permissions, settings will vary. Basically you try to publish, read error, add permission, rinse and repeat. Many tutorials just tell you to use admin account but I don't like that. Here's my IAM role policy but you'll need to adapt it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DeleteUserPool",
                "cognito-idp:AddCustomAttributes",
                "iam:UntagRole",
                "iam:TagRole",
                "iam:CreateRole",
                "iam:AttachRolePolicy",
                "iam:PutRolePolicy",
                "cloudformation:CreateChangeSet",
                "apigateway:DELETE",
                "cognito-idp:DescribeUserPool",
                "iam:PassRole",
                "iam:DetachRolePolicy",
                "cloudformation:DescribeStackEvents",
                "iam:DeleteRolePolicy",
                "apigateway:PATCH",
                "cloudformation:DescribeChangeSet",
                "apigateway:GET",
                "cloudformation:ExecuteChangeSet",
                "iam:GetRole",
                "apigateway:PUT",
                "cognito-idp:UpdateUserPoolClient",
                "iam:DeleteRole",
                "cognito-idp:ListTagsForResource",
                "cloudformation:DescribeStacks",
                "cognito-idp:CreateUserPoolClient",
                "apigateway:POST",
                "cognito-idp:UpdateUserPool",
                "iam:GetRolePolicy"
            ],
            "Resource": [
                "arn:aws:cognito-idp:<region>:<AccountID>:userpool/*",
                "arn:aws:iam::<AccountID>:role/<MyApp>*",
                "arn:aws:cloudformation:eu-west-1:<AccountID>:stack/jsbenchme*/*",
                "arn:aws:cloudformation:eu-west-1:<AccountID>:stack/aws-sam-cli-managed-default/*",
                "arn:aws:cloudformation:<region>:aws:transform/Serverless-2016-10-31",
                "arn:aws:apigateway:<region>::/restapis/*",
                "arn:aws:apigateway:<region>::/restapis"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:CreateUserPool",
                "cloudformation:GetTemplateSummary"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "events:*",
            "Resource": "arn:aws:events:iam::<AccountID>:rule/MyApp-*"
        },
        {
            "Effect": "Allow",
            "Action": "lambda:*",
            "Resource": "arn:aws:lambda:iam:<region>:<AccountID>:function:<MyLambdaFunction>-*"
        }
    ]
}

Organizing folders and files

I decided to put my AWS Serverless app templates into AWS subfolder of my project. Sam will create folder structure like this.

Run sam init and choose AWS Quick Start Templates, runtime (mine was Node) and Hello World Example. Folder structure like this will be created by Sam:

AWS
 |-- events -> where events for testing go
 |-- hello-world -> where your code goes (Lambda in my case)
 `- template.yaml -> Sam template to deploy resources through CloudFormation

First, I renamed hello-world with Lambda and put my Lambda code there, deleted some files too.
Next step, which might not be as obvious at start, is splitting up template.yaml in different files. If you have many services (especially API), file becomes unmanageable very fast. so I created folder ./templates for my templates expecting to merge them easily when the time comes to build.

Issue #1 - merging AWS Sam templates

It turns out there were several issues here. First about ability of Sam to merge files, then with some references requiring resources to be in the same template and last (but not least for sure) a lot of issues I had in my API Gateway definition yaml with references to other properties. More on that later. Enough to say for now that I solved all those issues with a small tool json-refs. So, I just did npm i -D json-refs, split template in multiple files, put them in ./templates folder and now my entry point template ./templates/main.yaml looks something like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    My App

Globals:
  Function:
    Timeout: 10

Resources:
  MyAppLambdaRole:
    $ref: 'LambdaAPIhandler.yaml#/Role'
  MyAppLambdaAPIhandler:
    $ref: 'LambdaAPIhandler.yaml#/Handler'
  MyAppDynamoDB:
    $ref: 'DynamoDB.yaml'
  MyAppAPIGateway:
    $ref: 'APIGateway.yaml#/APIGateway'
  MyAppFunctionLambdaPermission:
    $ref: 'APIGateway.yaml#/LambdaInvocationPermission'
  MyAppCognitoUserPool:
    $ref: 'CognitoUserPool.yaml#/Pool'
  MyAppCognitoClient:
    $ref: 'CognitoUserPool.yaml#/Client'

This is much neater and manageable. But to make it work, we need to setup script to use json-refs and merge those files into ./template.yaml. I decided to stick with Bash...

json-refs build script

We need to point json-refs to ./templates/main.yaml and it will render all those $refs and produce merged ./template.yaml for Sam to use.
I started by creating another folder ./templates/build, (also put it in .gitignore). Then created a Bash script ./build-deploy.sh which started like this:

    echo '[build] deleting build files'
    rm -r ./template/build/*

    echo '[build] copying templates to build folder'
    cp -r ./template/*.yaml ./template/build/

    echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
    ./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml

    echo '[build] cleaning up template.yaml'
    sed -i 's/T00:00:00.000Z//g' ./template.yaml

It's all quite self-explanatory, except maybe the last line where I remove T00:00:00.000Z which gets appended to version-dates of the Sam template. This might be unnecessary, but I like to keep thing as they are in AWS docs and versions don't have timestamps there.

Writing AWS resource definition yaml files

Again, relatively simple. AWS Docs are your best friend. It goes slow but steady. Some issues along the way though...

Issue #2 - json-refs complains about custom tags like !Ref, !Sub...

Very easy fixable once you find out that !Ref can be replaced with Ref:, !Sub with Fn:Sub and !GetAtt with Fn::GetAtt. They are identical but your new build script will not complain. Just don't forget colon (:) after those.

Issue #3 - API Gateway definition gets huge and has repetitive entries, in some cases (inside `DefinitionBody`) Ref does not work

My biggest fear was to write yaml for all API Gateway paths and methods, knowing how much time I spent on it in AWS Gateway UI. Exporting Swagger and pasting it in template required some changes. Some references were broken and I failed to make them work. It might be possible, but since I already had json-refs in place, I decided to use that, it seemed easier. At the end, it turned out writing API in yaml was easier. So much stuff can be reused, just define bits and pieces within components in your yaml and reuse them with $ref. I've put individual method responses to components, lambda invocation properties, even complete repetitive response objects for many paths. Just as an illustration, it can look like this now (shortened!):

APIGateway:
  Type: AWS::Serverless::Api
  Properties:
    Name: MyAPIGateway
    DefinitionBody:
      openapi: "3.0.1"
      path:
        "/me":
          get:
            x-amazon-apigateway-integration:
              $ref: "#/components/x-amazon-apigateway-integrations/default-integration"
            responses:
              $ref: "#/components/responses/defaultResponseGroup"

# components parsed by json-refs
components:
  responses:
    defaultResponseGroup:
      "404":
        description: "404 response"
        headers:
          $ref: "#/components/headers/default"
        content:
          $ref: "#/components/content/default"
      "200":
        description: "200 response"
        headers:
          $ref: "#/components/headers/default"
        content:
          $ref: "#/components/content/default"
  headers:
    default:
      Access-Control-Allow-Origin:
        schema:
          type: string
  content:
    default:
      application/json:
        schema:
          $ref: "#/components/schemas/Empty"

Of course, my definition has many paths, methods and responses, but putting some default properties and grouping them into components made it really easy to write API, easier than using API Gateway UI!

Issue #4 - circular references

Well, I let Sam create unique names for my resources so their Arns are created when deployed. But I need LambdaFunction to reference LambdaFunctionRole.Arn and LambdaFunctionRole to reference LambdaFunction.Arn. Sam cannot create both at the same time, they depend on each other. AWS docs have a good article on this and on more complex scenarios, but my solution was to name my LambdaFunction manually, so I can refer to it by name. It can be useful later in deployment process for other things too. Read on. Also check DependsOn CloudFormation attribute docs, this will help ordernig creation of resources to avoid similar issues in referrences.

Automating build & deployment to separate stages (test and prod)

My initial impression was that I'll use stageName and alias properties of some resources to build prod and test stages but it turns out best practice is to separate stages into different stacks. Actually, many people suggest using different accounts but that's not always practical. So, I decided that each stage will have it's own stack. I did use stageName too to make it even more clear which resource belongs where, although it seems unnecessary because we will make stackName also contain stage name to differentiate different stacks.
General idea is to use single code base for both prod and test stage, but before the build process I need to be able to decide which stage I'm going to build and publish. So there must be a way to make template(s) dynamic, to make certain properties changeable from command line. CloudFormation parameters and Sam's --parameter-overrides argument is what we need. First, I added parameterss to my ./templates/main.yaml:

Parameters:
  parAccountId:
    Type: String
    Description: AWS Account Id
    Default: ""
  parRegion:
    Type: String
    Description: AWS Region to deploy to
    Default: ""
  parStackName:
    Type: String
    Description: The name of the stack
    Default: "myStack-test"
  parStage:
    Type: String
    Description: "The name of the stage, must me one of: test, prod"
    AllowedValues:
    - test
    - prod
    Default: "test"

... then those parameters need to be used in templates in place of hard-coded ones (i.e. ${parStackName}).
Now is time to (maybe) make a step back. I'm actually editing this article after running into problems because I haven't done so. So, that step is to make naming convetion for your resources. Trust me, it will make your life easier if you just name all the resources manually, following that naming convention. Mine is:
<project-name>-<resource-name>-<stack>-<stage>

Go through all templates and implement this dynamic naming. Don't foget your sam-cli IAM role that you had to create at the beginning. This naming convention is going to make it a whole lot easier to avoid issues and resorting to '*'. Remember that Lambda function? Now part of its definition will look like:

  Type: AWS::Serverless::Function
  Properties:
    FunctionName:
      Fn::Sub: "MyProject-MyAppLambdaAPIhandler-${parStackName}-${parStage}"

Now, we can use sam build and sam deploy to build and deploy our resources. We will use --parameter-overrides argument to send those parameters to CloudFormation and they will affect template there (note! local template.yaml will NOT be changed based on params! This will happen on CloudFormation):

    sam build --build-dir .aws-sam/build \
                --template ./template.yaml \
                --region "${region}" \
                --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"

    sam deploy --template ./template.yaml \
              --no-fail-on-empty-changeset \
              --confirm-changeset \
              --tags stack=${stackName} stage=${stage} \
              --region "${region}" \
              --stack-name "${stackName}" \
              --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
              --capabilities CAPABILITY_NAMED_IAM

Of course, we will put this in our ./build-deploy.sh script which will take arguments and build those parameters with them, from command line. You can check it in the final script source at the end of the article.

Issue #5 - Sam requires S3 bucket to deploy

Well, actually this S3 bucket is made automatically, but ONLY in "guided" deploy mode which is activated by --guided argument of sam-build. In this mode, sam asks questions about some deployment parameters and creates S3 bucket auto-magically, then stores your choices to ./samconfig.yaml for later (add this file to .gitignore!). So, in our final ./build-deploy.sh script we will check for existence of this file and if it is not there, we will start deploy in "guided" mode. Just check the final code for details.

Issue #6 - Sam ignores --capabilities argument when in "guided" mode

We need --capabilities CAPABILITY_NAMED_IAM in my implementation and we do pass it to sam build as an argument. However, if it is in "guided" mode, it ignores --capabilities argument and asks you to enter the value manually (or uses wrong default). "Workaround" is also implemented in final ./build-deploy.sh.

Outputs

You will want to know some parameters created by AWS, which are not known before deploy is done. In example your API Gateway URI. We don't want anything hard-coded so there must be a way to fetch those after build & deploy. Options to the rescue. Again in ./main.yaml I added this:

Outputs:
  apiURL:
    Description: "production stage API URL"
    Value:
      Fn::Sub: "https://${MyAppAPIGateway}.execute-api.${parRegion}.amazonaws.com/${parStage}/"

This will create apiURL output visible in AWS CloudFormation UI. But how to access it from our build script and make it available to other parts of our app (in my case frontend React app)? Again, with Sam:

aws cloudformation describe-stacks --stack-name ${stackName} \
    --query 'Stacks[0].Outputs[?OutputKey==`apiURL`].OutputValue' \
    --output text

All we need to do now is to include this in our ./build-deploy.sh and add some code to save those values into a config file readable to our app (don't forget to .gitignore it too). Also don't forget to somehow differentiate same variables for different stages! Because ApiURL for prod stage and test stage should both be saved. You can see it all in final ./build-deploy.sh code where I also added more code to offer some help and arguments management:

#!/bin/bash

###################################################################################
#
# Bash script do build & deploy to different stages
#
###################################################################################

show-help () {
    echo 
    echo "--------------------------------"
    echo " AWS deployment script "
    echo "--------------------------------"
    echo 
    echo "usage: deploy.sh [options] <action>"
    echo 
    echo "ACTION:"
    echo "    build           Builds AWS app and stores template to ./template.yaml (Default action)"
    echo "    deploy          Deploys AWS app to CloudFormation using ./template.yaml"
    echo "    build-deploy    Builds and deploys in single step"
    echo 
    echo "OPTIONS:"
    echo "  mandatory arguments:"
    echo "    --accountId=   AWS account id to deploy to CloudFormation with"
    echo "    --region=       AWS region to deploy to"
    echo "    --stackName=   stack name"
    echo "    --stage=STAGE  stack stage name to deploy to"
    echo "                   STAGE is one of: prod, test"
    echo 
}

check-arguments () {
    # action arg default
    if [ -z "$action" ]
        then action = "build"
    fi

    # check empty mandatory args
    if [ -z "$accountId" -o -z "$region" -o -z "$stackName" -o -z "$stage" ]
        then echo "Error! Mandatory arguments missing"
    else
        # check format of args
        if [ "$stage" != "prod" -a "$stage" != "test" ]
        then
            echo "Error! --stage needs to be one of: prod, test"
            exit 3
        else
            return
        fi
    fi
    # output error specs
    if [ -z $accountId ]
        then echo " --accountId is a mandatory argument"
    fi
    if [ -z $region ]
        then echo " --region is a mandatory argument"
    fi
    if [ -z $stackName ]
        then echo " --stackName is a mandatory argument"
    fi
    if [ -z $stage ]
        then echo " --stage is a mandatory argument"
    fi
    exit 2
}

aws-build () {
    echo '[build] deleteing build files'
    rm -r ./template/build/*

    echo '[build] copying templates to build folder'
    cp -r ./template/*.yaml ./template/build/

    echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
    ./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml

    echo '[build] cleaning up template.yaml'
    sed -i 's/T00:00:00.000Z//g' ./template.yaml
}

sam-build () {
    sam build --build-dir .aws-sam/build \
                --template ./template.yaml \
                --region "${region}" \
                --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"
}

sam-deploy () {
    # check if samconfig.toml exists. If not, run --guided to autocreate S3 bucket and remember it
    if [ -f "./samconfig.toml" ]
        then isGuided=""
        else
            isGuided="--guided"
            printf "This is the first deploy, Sam will be started in --guided mode.\n \
                    NOTE! \n\
                    Please accept default params except those: \n\
                        Allow SAM CLI IAM role creation [Y/n]:   Select N! \n\
                        Capabilities [['CAPABILITY_IAM']]:   Enter: CAPABILITY_NAMED_IAM \n\
                    This is to avoid a bug(?) in Sam which (only on guided run) uses default capabilities instead of ones passed as an argument. \n\
                    You can also leave defaults and (after erroring out) rerun deployment again. Secontd time it will use passed arguments and run ok. \n\
                    \n"
            read -p "Press ENTER to continue."
    fi

    sam deploy "${isGuided}" \
              --template ./template.yaml \
              --no-fail-on-empty-changeset \
              --confirm-changeset \
              --tags stack=${stackName} stage=${stage} \
              --region "${region}" \
              --stack-name "${stackName}" \
              --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
              --capabilities CAPABILITY_NAMED_IAM
}

save-outputs () {
    echo "getting outputs and storing them in configs..."
    apiURL=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
        --query 'Stacks[0].Outputs[?OutputKey==`apiURLprod`].OutputValue' \
        --output text)

    cognitoClientId=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
        --query 'Stacks[0].Outputs[?OutputKey==`cognitoClientId`].OutputValue' \
        --output text)

    if [ -z "$apiURL" -o -z "$cognitoClientId" ]
    then
        echo "Error! Error getting output params $apiURL $cognitoClientId"
        exit 5
    else
        echo '{"apiURL": "'"$apiURL"'", "cognitoClientId": "'"$cognitoClientId"'"}' > ./options-${stage}.json
    fi
}


while [ $# -gt 0 ]; do
    case "$1" in
        --accountId=*) accountId="${1#*=}"
        ;;
        --region=*) region="${1#*=}"
        ;;
        --stackName=*) stackName="${1#*=}"
        ;;
        --stage=*) stage="${1#*=}" #allowed stage names are limited in template.yaml param definition, not here
        ;;
        build) action="build";;
        deploy) action="deploy";;
        build-deploy) action="build-deploy";;
        *)  echo 
            echo "Error: invalid argument: $1"
            echo "Use --help for list of arguments"
            show-help
            exit 1
    esac
    shift
done

check-arguments
echo "Running $action command of stack: $stackName, stage: $stage to region: $region using AWS accountID: $accountId"


if [ "$action" == "build" ]
    then
        aws-build
        sam-build
fi

if [ "$action" == "deploy" ]
    then
        sam-deploy
        save-outputs
fi

if [ "$action" == "build-deploy" ]
    then
        aws-build
        sam-build
        sam-deploy
        save-outputs
fi

Now all that is left is to use our script in package.json to build&deploy (and/or other scenarios) like:
./build-deploy.sh --accountId=012345678 --region=eu-west-1 --stackName=MyApp-test --stage=test build-deploy