DEV Community: Georgi Tenev

Sibling Docker containers during BitBucket Pipelines - A Kafka use-case

Georgi Tenev — Sun, 08 Mar 2020 23:05:16 +0000

In this article I will share my team's experience with setting up an infrastructure on BitBucket Pipelines to facilitate integration tests.

I will go over the challenges we faced when orchestrating our service's external dependencies as docker containers and making them communicate with each other both locally and during the CI integration tests.

Specifically, I will outline the docker-in-docker problem, how to solve it both locally, locally within docker & in CI and how to actually put it all together via docker-compose.yml & and a bit of Bash.

Our particular experience was with Kafka, but the broad ideas can also be applied to other stacks.

Background

For our current (greenfield) project my team is developing an application which is processing events from a Kafka stream. To ensure the velocity of the team doesn't decrease with the advancement of the project, we wanted to setup a suite of integration tests that will run on each commit during the CI testing phase within BitBucket Pipelines.

We had some specific requirements on how to ensure we can easily run the tests as often as possible - not only during the CI phase, but also locally:

Running the tests straight from our IDEs without too much wizardry was a must - PyCharm lets you click Run on a specific test which is like the best right :) Having a proper debugger is quite integral to us.
When running the tests locally, we wanted to be able to execute them directly via a native Python interpreter but also from within a container which resembles the one which will run during CI. PyCharm lets you use an interpreter from a docker container - we do this when we want to run our tests in a CI-like container and still have a proper debugger. For normal development/testing we prefer to use the native Python interpreter because of smoother development experience.
Have as close to prod environment as possible both locally and in CI. I.e. it must be easy to spawn the external dependencies and be able to have full control over them regardless of the environment.

Before we continue, I will reiterate the three environment we'll be addressing:

local - dev laptop using the native Python interpreter (via pyenv + pipenv)
local docker - again on dev laptop but python is within docker
CI - BitBucket Pipeline build which runs our scripts in a dedicated container

The tech stack

Our service is a Python 3.7 app, powered by Robinhood's Faust asynchronous framework. We use Kafka as a message broker. To ensure the format of the Kafka messages produced and consumed we use AvroSchemas which are managed by a Schema Registry service.

Project structure

Here's the project structure, influenced by this handy faust skeleton project:

./project-name
  scripts/
    ...
    run-integration-tests.sh
  kafka/ # external services setup
    docker-compose.yml
    docker-compose.ci.override.yml
  project-name/
    src/...
    tests/...
  bitbucket-pipelines.yml 
  docker-compose.test.yml
  ...

The kafka/docker-compose.yml has the definitions of all the necessary external dependencies of our main application. If run locally via cd kafka && docker-compose up -d, the containers will be launched and they will be available on localhost:<service port> - we can then run via our native Python the app and tests against the containerised services.

 # kafka/docker-compose.yml
 version: '3.5'
services:
  zookeeper:
    image: "confluentinc/cp-zookeeper"
    hostname: zookeeper
    ports:
      - 32181:32181
    environment:
      - ZOOKEEPER_CLIENT_PORT=32181
  kafka:
    image: confluentinc/cp-kafka
    restart: "on-failure"
    hostname: kafka
    healthcheck:
      test: ["CMD", "kafka-topics", "--list", "--zookeeper", "zookeeper:32181"]
      interval: 1s
      timeout: 4s
      retries: 2
      start_period: 10s
    container_name: kafka
    ports:
      - 9092:9092 # used by other containers launched from this file
      - 29092:29092 # use outside of the docker-compose network
    depends_on:
      - zookeeper
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:32181
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT_HOST://localhost:29092,PLAINTEXT://kafka:9092
      - KAFKA_BROKER_ID=1

  schema-registry:
    image: confluentinc/cp-schema-registry
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - kafka
      - zookeeper
    ports:
      - "8081:8081"
    environment:
      - SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:32181
      - SCHEMA_REGISTRY_HOST_NAME=schema-registry
      - SCHEMA_REGISTRY_DEBUG=true
      - SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081

A peculiarity when configuring the kafka container is the KAFKA_ADVERTISED_LISTENERS - for a given port you can specify a single listener name (e.g. localhost for port 29092). Kafka will accept connections on this port if the client used kafka://localhost:29092 as the URI (plus kafka://kafka:9092, for the example above). Setting 0.0.0.0 instead of localhost causes kafka to return an error during launch - so you need to specify an actual listener hostname. We will see below why this is important.

CI BitBucket Pipelines & Sibling Docker containers

As per the Bitbucket documentation, it's possible to launch external dependencies as containers during a Pipeline execution. This sounded appealing at first but it meant it would lead to difficulties when setting up the test infrastructure locally as the bitbucket-pipelines.yml syntax is custom. Since it was a requirement for us to be able to easily have the whole stack locally and have control over it, we went for the approach of using docker-compose to start the dockerised services.

To configure the CI we have a bitbucket-pipelines.yml with, among others, a Testing phase, which is executed in a python:3.7 container:

definitions:
  steps:
    - step: &step-tests
        name: Tests
        image: python:3.7-slim-buster
        script:
          # ...
          - scripts/run-integration-tests.sh # launches external services + runs tests. 
        ...

The run-integration-tests.sh script will be executed inside a python:3.7 container when running the tests via docker (locally & on CI) - from this script we launch the kafka & friends containers via docker-compose and then start our integration tests.
However, doing so directly would mean that we will have a "docker in docker" situation which is considered painful. By "directly" I mean launching a container from within a container.

A solution to avoid "docker in docker" is to connect to the host's docker engine when starting additional containers within a container instead of to the engine within the container. By connecting to the host engine, sibling containers will be started, instead of child ones.

The good news is that this seems to be done out of the box when we are using BitBucket pipelines! So it's a concern just when we're running locally in docker.

The pain & the remedy

There are two main problems:

how to use the host Docker engine from within a container instead the engine within the container
how to make two sibling containers communicate with each other
how to configure our tests & services so that communication can happen in all three of our environments

Use host Docker engine

In the root of our project we have a docker-compose-test.yml which just starts a python container and executes the scripts/run-integration-tests.sh.

version: '3.5'
services:
  app:
    tty: true
    build:
      context: .
      dockerfile: ./Dockerfile-local # installs docker, copies application code, etc. nothing fancy
    entrypoint: bash scripts/run-integration-tests.sh
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock # That's the gotcha

When the above container launches, it will have docker installed and the docker commands we run, will be directed to our host's docker engine - because we mounted the host's socket file in the container.

To start this test container we actually combine both the kafka/docker-compose.yml & docker-compose-test.yml - via docker-compose --project-directory=. -f local_kafka/docker-compose.yml -f ./docker-compose-test.yml -f ... <build|up|..... This is just an easy way to extend the compose file with all the kafka services with an additional service. We have this in a Makefile to save ourselves from typing too much. E.g. with the Makefile snippet below we can just do make build-test to build the test container.

command=docker-compose --project-directory=.  -f local_kafka/docker-compose.yml -f ./docker-compose-test.yml
build-test:
    $(command) build --no-cache

Sibling inter-container communication

When running the containers locally

It's cool that we can avoid the docker-in-docker situation but there's one "gotcha" we need to address :)

How to communicate from within one sibling container to another container?

Or in our particular case, how to make the python container with our tests connect to the kafka container - we can't simply use kafka://localhost:29092 within the python container as localhost will refer to, well.., the python container host.

The solution we found is to go through the host and then use the port of the kafka container which it exposed to the host.

In newer versions of Docker, from within a container, host.docker.internal will point to the host which started the container. This is handy because if we have the kafka running on the host as a container and exposing its 29092 port, then from our python tests container we can use host.docker.internal:29092 to connect to the sibling container with kafka! Good stuff!

So in essence:

when running the tests from the host's native python we can target localhost:29092
and when running the tests from within a container, we need to configure them to use host.docker.internal:29092.

When running the containers on BitBucket

Now there's another gotcha, as one might expect. When we are in CI and we start our containers, host.docker.internal from within a container does not work. Would have been nice if it did, right... The good thing is that there is an address that we can use to connect to the host (the bitbucket "host", that is).

I found it by accident when inspecting the environment variables passed to our build - the $BITBUCKET_DOCKER_HOST_INTERNAL environmental variable keeps the address that we can use to connect to the BitBucket "host".

Now the above is actually the significant findings that I wanted to share.
In the next section I will go over how to actually put together the files/scripts so that it all glues together in all of our three envs.

Gluing it all together

Given the above explanation, the trickiest part, in my opinion, is how to configure the kafka listeners in all three envs & how to configure the tests to use the correct kafka address (either localhost/host.docker.internal/$BITBUCKET_DOCKER_HOST_INTERNAL). We need to configure the kafka listeners accordingly, simply because our test kafka client will use different addresses to connect to kafka, thus the kafka container should be able to listen on the corresponding address.

To handle the kafka listener part we make use of the docker-compose files extend functionality. We have the "main" kafka/docker-compose.yml file which orchestrates all containers and configures them with settings that will work fine for when using the containers from our local python via localhost. We also added a simple docker-compose.ci.override.yml with the following content:

version: '3.5'
services:
  kafka:
    environment:
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT_HOST://${BITBUCKET_DOCKER_HOST_INTERNAL:-host.docker.internal}:29092,PLAINTEXT://kafka:9092

This file will be used when starting the kafka & friends when running tests within local docker & CI - if the $BITBUCKET_DOCKER_HOST_INTERNAL is set, then its values are going to be used - we are in CI then. If it's not set, the default host.docker.internal is used - meaning we are in local docker env.

Our tests/app are configured via env vars. Our run-integration-tests.sh looks something like this

pip install --no-cache-dir docker-compose && docker-compose -v
docker-compose -f local_kafka/docker-compose.yml -f docker-compose-test-ci.override.yml down
docker-compose -f local_kafka/docker-compose.yml -f docker-compose-test-ci.override.yml up -d

# install app dependancies

# wait for kafka to start
sleep_max=15
sleep_for=1
slept=0
until [ "$(docker inspect -f {{.State.Health.Status}} kafka)" == "healthy" ]; do
  if [ ${slept} -gt $sleep_max ]; then
    echo "Waited for kafka to be up for ${sleep_max}s. Quitting now."
    exit 1
  fi
  sleep $sleep_for
  slept=$(($sleep_for + $slept))
  echo "waiting for kafka..."
done
# !! figure out in which environment we are running and configure our tests accordingly
if [ -z "$CI" ]; then
  if [ -z "$DOCKER_HOST_ADDRESS" ]; then
    echo "running locally natively"
  else
    echo "running locally within docker"
    export KAFKA_BOOTSTRAP_SERVER=host.docker.internal:29092
    export SCHEMA_REGISTRY_SERVER=http://host.docker.internal:8081
  fi
else
  echo "running in CI environment"
  export KAFKA_BOOTSTRAP_SERVER=$BITBUCKET_DOCKER_HOST_INTERNAL:29092
  export SCHEMA_REGISTRY_SERVER=http://$BITBUCKET_DOCKER_HOST_INTERNAL:8081
fi
pytest src/tests/integration_tests

The script can be used in all of our three envs - it will set the KAFKA_BOOTSTRAP_SERVER (our tests' configuration) env var accordingly.

FYI In the Makefile example couple of paragraphs ago, I've omitted the ci.override.yml file for brevity.

Thanks for reading the article - I hope you found useful information in it!
Please let me know if you know better ways to solve the problems from above! I couldn't find any, thus wanted to save others from the pain I went through :)

What are the skills a junior DevOps should have

Georgi Tenev — Fri, 23 Aug 2019 09:27:46 +0000

Hi folks! :)
I've been working as a DevOps for nearly two years. I was offered to teach a small group of students a DevOps course. The outcome for them should be that they are hireable at a junior level.
What skills and technologies you think are a must for them to know and understand?

Cheers :)

Protecting your Git branches on AWS CodeCommit

Georgi Tenev — Tue, 19 Mar 2019 18:30:31 +0000

Borovets, Rila mountain, Bulgaria

This post gives a quick recipe on how to enable only selected users to merge/commit into CodeCommit Git branches.

The problem

Let's consider a common CD/CI scenario:
1) you have a team which pushes to a Git repo with important code.
2) when ready for a deploy, someone merges dev into master
3) the merge event triggers a preconfigured deployment pipeline which would package the code from the repo and then delpoy it to production

The above is very nice and all, but care must be taken when it comes to who should be allowed to merge into the production branch.

The solution

(jump straight to the sample IAM policies)

Since we are in the context of AWS, the solution lies in using the IAM service (Identity and Access Management).

Here are the main steps of the solution:

Have a generic group for all developers, say Dev-Group. All developers within your team belong to this group. Generic CodeCommit actions are allowed - e.g. pull, clone, etc.
Also attach to the Dev-Group a policy which specifies the repositories which we want to protect via the NotResource. For all other repos developers can have full access, if so desired.
Have a second IAM Group - e.g. ImportantRepo-PowerUser - it has a policy which enables the merge/commit actions for the ImportantRepo repo.
Add the selected few developers which should be able to trigger a build to this ImportantRepo-PowerUser group

Why bother writing this article - well, it's not that obvious how to implement the above. The tricky part is the evaluation logic of IAM. Due to it, the naïve solution below won't work:

all devs in Dev-Group, the selected few in SelectedFew
use a Deny statement in a policy in Dev-Group to deny devs merge into master
attach a policy to SelectedFew which allows its users to merge into master

If in the Dev-Group you add a statement which explictly forbids (with a Deny) users of the group to merge into master, then it's not possible to allow these actions in a different policy - so it won't be possible to allow our selected few developers to merge.

This is where the NotResource comes in handy. With it you can exclude the protected repositories from a given statement without explicitly using the Deny action. I.e. you can give full perms to all repositories excluding the protected ones. Since we haven't explicitly Deny-ied actions on the protected repositories, we can later add explicit Allow in a different policy - in a policy attached to the SelectedFew group.

Show me the code

dev-group-generic-policy

We need the Dev-Group which gives generic permissions to all devs and also lists the protected repositories.

// the policy atached to the Dev-Group
{
  "Version": "2012-10-17",
  "Statement": [
    // generic permissions for all devs
    {
      "Effect": "Allow",
      "Action": [
        "codecommit:Get*",
        "codecommit:GitPull",
        <whatever you need but not Push/Merge/etc.>
      ],
      "Resource": "*"
    },

    // the gotcha - in the NotResource put the important repos whose branches you want to protect
    {
      "Effect": "Allow",
      "Action": [
        "codecommit:GitPush",
        "codecommit:DeleteBranch",
        "codecommit:Merge*"
      ],
      "NotResource": [
        "arn:aws:codecommit:*:*:<important-repo-name-1>",
        "arn:aws:codecommit:*:*:<important-repo-name-2>"
      ]
    },

    // still allow all devs to push to non-production branches of the imporant repos
    {
      "Effect": "Allow",
      "Action": [
        "codecommit:GitPush",
        "codecommit:DeleteBranch",
        "codecommit:Merge*"
      ],
      "Resource": [
        "arn:aws:codecommit:*:*:<important-repo-name-1",
        "arn:aws:codecommit:*:*:<important-repo-name-2"
      ],
      "Condition": {
        "StringNotEquals": {
          "codecommit:References": [
            "refs/heads/master",
            "refs/heads/prod",
            "refs/heads/Stg"
          ]
        }
      }
    }
  ]
}

important-repo-1-power-users

A group for the power users for a specific important repo.

// the policy attached to the group for users that can push to all branches for a given important repo 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "codecommit:GitPush",
        "codecommit:DeleteBranch",
        "codecommit:Merge*"
      ],
      "Resource": [
        "arn:aws:codecommit:*:*:<important-repo-name-1>",
      ]
    }
  ]
}

To make the solution more generic, you can use some naming convention for the repositories - i.e. all repo names starting with xyz- are considered important and only users in group xyz-power-users can merge into the prod branch of these repos.

Creating a native mobile app with NativeScript — tips and tricks

Georgi Tenev — Tue, 20 Mar 2018 10:56:22 +0000

Couple of months ago I decided to write an app that I can use to keep track of my expenses. I wanted a contemporary-looking app, with no ads, for free. I wanted to give mobile app development a try. In January I did an AWS training, so I also wanted to apply what I’ve learned, for a backend API.

In this post I will share some parts of the process of designing, implementing and deploying the Para app, together with some of the difficulties and bugs I’ve encountered along the way.

The aim is to share my findings and ultimately help people that are new to mobile app development, and in particular to the NativeScript platform.

I will focus on the app itself. I’ll talk about the backend API in another post. The API is developed with Python + Flask & DynamoDB, deployed on AWS via Zappa.

The code for the mobile app (TypeScript) and the backend (Python) is open-sourced.

The article starts with discussion about Firebase Auth and how I’ve used it. If you only care about the NativeScript-specific gotchas — scroll down to the Helpful NativeScript-specific plugins and gotchas section.

Introduction

When planning how to make the app, I spent some time researching platforms for cross-platform mobile app development. I wanted to write the code once, and then be able to run Para natively on both Android & iOS. The main candidate platforms were React Native and NativeScript. I already had some experience with JavaScript/TypeScript, but none with React*. NativeScript lets you use pure TypeScript and it’s developed by Telerik, which was originally a Bulgarian company. Also, it seemed that the majority of the comparison articles I found on Google, were suggesting that NativeScript is better.

All of this was enough for me to try NativeScript first.

Reading this post will definitely be more effective if you’ve already read the official NativeScript Getting Started guide.

* I know, I know, it’s next on the list.

Definition of done

App to help me keep track of my expenses.
Easy to add new expenses and view existing ones.
Supports email & social login.
Offers statistics (“How much I’ve spent this week/month”).
Costs me 0$ per month to run the app. Given that it’s published on an app store and has more users than me and my dad.
Runs natively on Android and iOS.
Can handle 500 users that simultaneously use the app.
Can handle a user which uses the app on multiple devices.

10 000 ft. overview

The app is responsible for:

validating & collecting user input (when adding new expenses / updating an expense)
displaying the user’s expenses + statistics

The API is responsible for:

validate & store expenses to a database
retrieve expenses and stats about them from the database

Firebase is responsible for:
*storing user sensitive auth data

auth related activities like email confirmation, resetting passwords, etc.

Both the app and the API share the same representation of what is an expense (declaratively, by using a json schema).

By design, no user credentials (emails, passwords) are stored in the database I manage (the DynamoDB).

Authentication and Firebase

The difficulty — how to authenticate the mobile app to the backend API with minimal effort, while offering an adequate solution.

I used Firebase for auth related activities in the app. Firebase identifies each user of a given project with an user uid (it’s a long string).

When a user logs in (either by entering email + password or via Facebook), his user uid is accessible from within the app code. Currently, in the context of the app, all expenses belong to the currently logged in user, thus the uid doesn’t appear in the schema of expense objects.

The uid is useful when the app communicates with the backend API though — because the API needs to perform CRUD operations in the context of the correct user.

Sending a uid directly would not be safe because impersonating users would be possible. Firebase offers the ability to generate ID Tokens. Here’s my mental model about them. It’s a JSON Web Token (JWT), a string, generated by the Firebase Client SDK (i.e. a SDK that runs on the phone). The string contains the currently logged-in user’s uid, the time the user logged in and some other user info (email, etc.), which Firebase has about the user. The nice thing about this string is that when our backend receives it, the server can verify the integrity of the string — i.e. if the string says that the user with uid XXXYYY sent the string — we can be certain that indeed it was this user that sent the string.

The general workflow is that when the phone makes an API request to our backend, the ID Token is included in a HTTP header. When the server receives the request, it decodes the token, by using the Firebase Admin SDK. Then the user uid is available to the server and it’s possible to process the request in the correct context.

Why I chose this approach? It releases me from the hassle of worrying about storing user emails, passwords, dealing with email confirmation, forgotten passwords, etc. It is a hassle. And since it’s easy to mess it up, I delegated that to Firebase. All I care about is ensuring I can show the login/signup page to my app users and once they’ve enter what they should, ask Firebase to give me info about the user (his uid and id token).

Another noteworthy aspect of using Firebase is having isolated environments.

I want to have completely separate development and production environments. Meaning, that I can develop locally and not touch the production Firebase project — which has all my users. Achieving this is a matter of creating a per-environment Firebase project and choosing the correct Firebase configuration file during app build (or “prepare”) time. I have the production file stored securely and can use it to make production app builds which are uploaded to the app/play store. I can share the development configuration with other collaborators so that they can develop locally.

Since the backend’s Firebase Admin SDK needs to have credentials to the same Firebase project, I store the credentials as an environmental variable — and the correct credentials are used for a given server environment. This is really handy because it’s not possible for a development app to access a production server — can happen if I misconfigure the app (e.g. point to the prod API when I develop locally). You can tell I don’t trust myself a lot :)

That’s it for the 10,000 ft. overview.

I will show some of the NativeScript-specific plugins I found useful and some problems / gotchas related to them.

Helpful NativeScript-specific plugins and gotchas

When using NativeScript, you can benefit from npm packages. E.g. I used the popular moment.js and underscore JavaScript libraries without any issues. Here’s a page with verified NativeScript-targeted packages. Here’s a page with overview of how to install and use them.

Just as a quick note, be careful when installing packages from npm — sometimes the packages assume they’ll run either in a browser or a Node.js environment — and can fail if ran within NativeScript (e.g. I saw a few packages failing to import process).

`nativescript-plugin-firebase`

Instead of using directly the official Firebase Android/iOS Client SDKs, I used the nativescript-plugin-firebase. The benefit of that is that I don’t need to write Java and Swift — I write TypeScript, once. As mentioned above, I use Firebase for all things that are auth — registering, signing in and logging out users. Changing their password and recovering their password are also made possible by the plugin. And most importantly, getting the user’s uid and ID Token.

The one thing that wasn’t immediately obvious to me was how to ensure that once the user logs in, he stays logged in, until he explicitly logs out. The problem was that if you login and then minimize the app and restore it after an hour, auto logging in will succeed, however the ID Token will be expired — tokens are only valid for an hour since their creation (i.e. last time you entered email + password and signed in). And if I try to send an expired token to the backend — the backend will bark.

The trick, that worked for me, is to use firebase.getAuthToken({forceRefresh: true}) when auto-signing in the user. This will force the creation of a fresh ID Token. Here’s an example usage. In the code I check if the user has already logged in, and if so, request a fresh token immediately. The login-page is the default first page of the app and it’s shown always when you open the app. The check from above ensures that if a user is logged in, he’ll be redirected to the “home” page directly.

`nativescript-pro-ui`

The pro-ui package (it’s actually a bundle of packages since a couple of weeks) provides reusable UI components.

I use it for three different things in my app — for the list of all expenses, for the side drawer and for a data-form to create/edit expenses.

`RadListView`

Why using the Pro UI’s RadListView is indeed rad? Because all I need to do to show the list of expenses from above is: say how each expense of the list should be displayed (i.e. given an expense data object, which attributes to show and where to put them). That’s done via XML, provide a collection of expenses to show (it’s called an ObservableArray and you can think of it as a beefed-up array that emits events when you add/delete to it)

The nice thing is that there’s a binding between the collection of expenses I provide and the RadListView — i.e. if I add/remove an expense to my collection, the RadListView will automatically update. This means I can only focus on making sure my array has the correct items and not on UI.

Apart from that, I get “free” pull-to-refresh and loading data in batches (i.e. 10 expenses per batch are loaded). To get the former two I only need to provide functions which should actually refresh the data / get the next batch of items from the API and update the items in the ObservableArray.

There’s an important gotcha here. When writing the XML markup for the “list all expenses” page, I wanted to show different things depending on whether the user has any expenses or not. If the user doesn’t have expenses — I show a message together with an “add new” button, if there are expenses — just show them. I implemented this using styles (think CSS) by

RadListView visibility=”{{hasItems ? ‘visible’ : ‘collapse’}}”. So I use some boolean variable hasItems which determines whether we see the list or not.

The trick is that one might think that since the collection of expenses we provide is of type ObservableArray, the .length property is also observed. Well, it seems it’s not. So binding to the expression expenses.length !== 0 is not possible. My workaround was to subscribe to the events of the ObservableArray and adjust the value of hasExpenses on each add/remove.

If the above doesn’t make a lot of sense, make sure you’ve read the section about binding in the documentation.

`RadSideDrawer`

The drawer is quite handy — you can put the hamburger on the top left of each screen and when you press it or when you slide from left to right, the side panel drawer appears.

It’s easy to have a lot of duplicated code this way though — i.e. on each page you want a side drawer, you write essentially the same code — the content of the drawer and then the content of the page.

I extracted the code for the drawer content into a separate component. I found this article really informative about how to extract NativeScript UI components and reduce duplication.

Frankly, I now have the equivalent of “import the side drawer” code in most of my pages which is sort of duplication as well. As a further refactoring I plan to extend the Page class to PageWithDrawer and use it as the root element of my pages.

`RadDataForm`

Oh, this one’s my favourite. It’s really handy. Essentially, you provide a plain old data object, and optionally description of the properties of the object, and you get a UI data form. You can register callbacks called during validation of user entered data. You choose when the data the user entered is committed to the input data object. You can also register a callback for when data is committed. You choose if committing input to the data object happens immediately or after the user has, say, pressed the Submit button.

You get different field types out of the box (e.g. text, number, email, etc.) — handy because you get some auto-validation and the proper keyboard is shown.

The easiest way to configure the visible fields, their type (text, number) and validators (MinLength, NonEmpty, etc.) is via XML markup.

<RadDataForm.properties>
    <EntityProperty name="name" displayName="Name" index="0" />
    <EntityProperty name="age" displayName="Age" index="1" >
      <EntityProperty.editor>
        <PropertyEditor type="Number" />
    <EntityProperty.validators>
          <RangeValidator minimum="1" maximum="150" /> 
        </EntityProperty.validators>
      </EntityProperty.editor>
    </EntityProperty>
</RadDataForm.properties>

I recommend reading all of the docs on RadDataForm if you plan to use it. It’s not long and it will save you a ton of time.

During development of the app, I found myself in a situation in which I needed very similar dataforms with the only difference being the action performed after the Submit is pressed. The prime example — creating a new expense or updating an existing one. From the perspective of the DataForm, in both cases the input data object has the same shape, with the difference that during updating an expense, the data object has actual data. When the user presses the Submit button of the form, a different API endpoint is called. But for the rest, the form is the same — validation, field types, etc.

Thankfully, when creating data forms, instead of using just XML markup to describe the properties, you can pass JSON. So instead of just passing the source data object and using XML, you pass the data object and a “metadata” JSON, which describes the properties instead. Here’s how the above XML can look as JSON metadata:

{
    "name":"name",
    "displayName":"Name",
    "index":0
},
{
    "name":"age",
    "displayName":"Age",
    "index":1,
    "editor":"Number",
    "validators":[
        {
            "name":"RangeValidator",
            "params":{
                "minimum":1,
                "maximum":150
            }
        }
    ]
}

The examples from the NativeScript’s ui-samples-repo, seem to all read the metadata from a JSON and pass it directly to the DataForm. What I did was to have a function which generates the JSON. This gave me a lot of flexibility when generating the metadata. There’s info in the docs on what you can and cannot do with JSON compared to via XML markup.

And now the gotcha! :) I spent good couple of days chasing a bug related to RadDataForm. Essentially I was validating the form manually and committing the input to the data object manually too. So after doing that I was accessing the data object’s properties because I needed their values. The bug was that if the property was marked as a Number or Decimal in the metadata of the form, getting its value doesn’t return the number but rather some object of unknown type. The workaround is to call the toString() of this object to get the string representation of the number and then convert it to a proper number. Nasty.

`i18n`

I only found out last week that it means internationalization :)

I wanted to make my app available in Bulgarian and English. I googled around for a NativeScript i18n package that is actively developed and came across — nativescript-localize.

The only gotcha here. When doing debug builds it’s ok to do L(“natural language in english”) and only define a translation in the non-english langs for the string — it will fallback to the english string automatically if needed. However, when doing a release build, the lint will bark. The article mentions some fixes, but as far as I can tell they are not applicable when using the nativescript-localize package because the package generates the string.xml on build time. In essence you might want to use non-user-friendly strings and provide natural language translations for them for each supported language.

I recently found out that the popular i18next JavaScript package also seems to work — probs because it makes very conservative assumptions about the runtime.

Using the library within TypeScript/JavaScript doesn’t deviate from the package’s suggested way. However, in NativeScript we want to translate string in XML pages too. The gotcha is setting the app resources (i.e. making a function available within XML). Doing it like this seems to work correctly:

app.setResources({
    ‘L’ : (…args) => i18next.t.apply(i18next, args)
});

This enables us to do stuff like:

<Label text=”{{‘translation for key is ’ + L(‘key’)}}”/>

`JSON Schema`

Apparently It’s not that easy to find a package to do validation of a JSON Schema in a NativeScript app.

After a lot of trial and error, I found a JavaScript package that works well. It’s called tv4. Frankly, I haven’t tried more advanced use-cases like using schemas from different files so I can’t say how it behaves there.

`Mocking in tests`

I like mocks a lot when testing. I tried to use different mocking frameworks. Most often the problem with the frameworks is that they make the assumption that the tests run in node/browser which is problematic in our case.

I had success only with using the built-in mocking mechanism of Jasmine. It does what I need, but if you know a different package — I’d be super curious to check it out :)

`Misc`

SideKick — if you like GUIs it can be helpful. For me the greatest benefit of it are the cloud builds — it lets me build for Android/iOS from my Linux host.
AWS Device Farm — I test my release builds of the app there. There’s a selection of iOS/Android devices. Not the most up-to-dated though. You get 1000/month within AWS’s free tier.

Conclusion

Overall, I enjoyed my experience developing the app. I enjoyed the fact that I can use npm packages and not reinvent the wheel for each problem. Haven’t tested the app on iOS yet (don’t have a dev account yet) but in theory it should work with no code changes.

I wish I started using hooks earlier — I use it to choose/change my app config based on whether I am building a development app or a release one. I also use it to make checks like “Is my API address in a correct format and is it an address I’ve whitelisted”, to get the current git sha and put it in the app config so I know the exact code that the app runs, etc.

Thanks for reading this. If you have any suggestions or remarks, please share them — I will be grateful.