DEV Community: Jameson

Pre-push Hooks

Jameson — Wed, 10 Apr 2024 21:56:27 +0000

What is a Hook?

Git has a few different extension points where you can execute arbitrary scripts. Two of the most well-known are the pre-commit and pre-push hooks. As discussed below, these hooks allow you to perform additional validation before completing a commit or push.

Why Not Just CI?

As a repository grows, you onboard more linting tools, test suites, and other validations. The goal is to catch regressions and bad practices as early as possible. CI/CD (e.g., GitHub Actions) provides a means to ensure that these validations are run on every PR and every commit before being merged, as an invariant.

However, CI/CD jobs can grow to be quite slow - hours, in some cases. For certain tasks that are quick to run, you'd probably prefer to run them on your local development machine before raising a PR.

For example, checking if the locally-changed code is well-formatted is one task that can be done fairly quickly, and makes sense to do before raising a PR.

On the other hand, running a full test suite may be best handled by the CI as a final check. Most changes aren't at risk of breaking unrelated tests, and the developer would like to work on other things while CI works in parallel.

Pre-commit vs. Pre-push

As with most things in Engineering, there is a trade-off between instrumenting your validations in a pre-commit vs. a pre-push hook.

The pre-commit hook will run much more often, and can potentially enable you to keep each commit in a valid state. However, if you need to invoke the project's build system to complete validation, it may start to become a nuisance and ultimately slow you down. Worse, if you're making surgical, atomic commits, the pre-commit hook might even mess up your tree.

Pre-push hooks don't have most of those problems and are the final opportunity to check code before it goes up for PR. One downside: you may end up with an additional "fix formatting"-type commit in your log this way. But, if you have GitHub configured to squash commits on merge, it doesn't matter much. Since pre-push hooks are run much less frequently, it also makes sense to perform some slightly longer validations in this hook.

In a pre-push hook, you have already established a lineage of commits that will be sent for PR. So, it is also a safer time to apply edits to the codebase, if you want to. For example, if the code in the push ref has bad formatting, we might like to format the code instead of just simply checking it. In a pre-commit hook, that might be disruptive to in-flight coding.

How to build a pre-push system

Various tools do provide their own integrations to Git hooks. For example, The Gradle ktlint Plugin offers a few tasks to register check/format tasks as a Git hook. But, as soon as you want to run multiple tasks in your hook, you'll probably wish you had better machinery.

So, let's state a few of our up-front goals. We want to:

Create an extensible script that will allow us to run multiple validation scripts during push;
Provide clear and actionable feedback to the user about what is happening with the push - did it succeed or not, and why not?;
Provide a simple way for developers on our team to install the pre-push scripts and get pertinent updates as improvements are added.

To solve the first problem, let's create a directory in our project called ./scripts/pre-push.d (inspired by the Linux/UNIX practice of splitting out config stubs in .d directories.) We'll plan to have a ./scripts/pre-push script that calls out to the stubs:

To solve the second problem, we'll be careful to fail the pre-push as soon as one of the stubs fails and to be thoughtful about the error message we emit if it does fail.

We can solve the third problem by creating a simple ./scripts/install-pre-push script that our teammates can run. Recall that there's no simple way to have hooks automatically added to .git/hook when a user clones a repository. So, you'll need to ask them to run this script when they're doing developer onboarding.

Our installation script will do a little cleanup in .git/hooks to ensure no old version of hooks will cause a conflict. Then, it will simply create a symbolic link from our ./scripts/pre-push script into .git/hooks. This way the developer will always run the latest pre-push script contents without having to re-run ./scripts/install-pre-push each time there's an update to its logic.

The installation script

The install-pre-push script looks as below. Note that it mostly does cleanup, and then creates a symbolic link.



#!/bin/bash

# Print a message and kill the script.
die() {
    echo "$@" 1>&2
    exit 1
}

# Finds the top of the repo.
find_git_repo_top() {
    local current_dir=$(pwd)

    # Loop until reaching the root directory "/"
    while [ "$current_dir" != "/" ]; do
        # Check if ".git" directory exists
        if [ -d "$current_dir/.git" ]; then
            echo "$current_dir"
            return
        fi

        # Move up one directory
        current_dir=$(dirname "$current_dir")
    done

    # If ".git" directory is not found
    echo "Git repository not found."
    exit 1
}

# Ask the user a yes/no question and await their response. Return 0 if
# they say yes (in some format).
await_yes_no() {
    read -r answer
    case "$answer" in
        [yY]|[yY][eE][sS])
            echo 0
            ;;
        *)
            echo 1
            ;;
    esac
}

# Delete whatever hooks may be active in the .git/hooks directory. This
# may include things like the old pre-commit hook we had been using
delete_existing_hooks_with_confirmation() {
    project_root="$1"
    hooks=$(find "${project_root}/.git/hooks/" -mindepth 1 ! -name "*.sample")
    echo "Found hook files: $hooks"
    echo "OK to delete? [Y/n]"
    if [ "$(await_yes_no)" -ne 0 ]; then
        die "OK; aborting."
    fi
    rm -f -r $hooks
} 

# Install the pre-push script and any hooks found under the ./scripts
# directory.
install_pre_push_hooks() {
    project_root="$1"
    echo "Installing scripts into .git/hooks ..."
    mkdir -p "${project_root}/.git/hooks"
    ln -s "${project_root}/scripts/pre-push" "${project_root}/.git/hooks/pre-push"
}

# Installs pre-push scripts after ensuring we're running from the
# directory root, and after cleaning up any old git hook scripts.
main() {
    project_root="$(find_git_repo_top)"
    delete_existing_hooks_with_confirmation "$project_root"
    install_pre_push_hooks "$project_root"
}

main

The pre-push script

The pre-push script is also pretty simple. Its primary job is to find the stub files in ./scripts/pre-push.d, and then send them the same arguments that Git sent to it.

For a pre-push script, git will send four pieces of information to the hook - and possibly multiple times, for each reference being pushed:

localname
localhash
remotename
remotehash

Most of the time, you'll run something like:



git push origin my-branch

In this case, remotename and remotehash won't be populated with meaningful info, and the four parameters will only arrive once.

It is possible, however, to run more complex push commands such as



git push origin my-branch-1:refs/heads/target-1 my-branch-2:refs/heads/target-2

This is useful for us to understand the meaning of the four arguments, but as we'll see later, we won't need them. Most of our practical hooks will only consider the localhash.



#!/bin/bash
#
# Don't write actual logic in this file. This file just fans out to
# .git/hooks/pre-push.d/<your_file>, so that we can add a number of
# checks into one place. This file aims to honor the original pre-push
# contract and fan it out to stub files.

# Finds the top of the repo.
find_git_repo_top() {
    local current_dir=$(pwd)

    # Loop until reaching the root directory "/"
    while [ "$current_dir" != "/" ]; do
        # Check if ".git" directory exists
        if [ -d "$current_dir/.git" ]; then
            echo "$current_dir"
            return
        fi

        # Move up one directory
        current_dir=$(dirname "$current_dir")
    done

    # If ".git" directory is not found
    echo "Git repository not found."
    exit 1
}

project_root=$(find_git_repo_top)
hooks=$(find ${project_root}/scripts/pre-push.d -type f ! -name "*.sw*" | sort)

# pre-push.d receives four pieces of information for each source-target
# push that may be in play. For exmaple, origin->refs/heads/origin, and
# my_kooll->refs/heads/my_kool_branch would cause two iterations of this
# while loop.
while read localname localhash remotename remotehash; do
    # For each set of push data, iterate over the hooks in alphabetical
    # order. Pass the hook data in using the same contract.
    for hook in $hooks; do
        echo "$localname $localhash $remotename $remotehash" | bash "$hook"
        RESULT="$?"
        if [ $RESULT != 0 ]; then
            exit "$RESULT"
        fi
    done
done

exit 0

A pre-push.d script

The more of these you see and write, you realize there are lots of edge cases. So let's consider our ultimate goal here.

The branch we want to keep in good condition is origin's main. In trunk-based development, we don't care so much about other branches. So as I mentioned above, let's not worry about the remotehash and remotename; let's always compare our local changes against origin/main.

Another concern is performance. Ideally, we'd like to use work avoidance to skip these validations entirely if there have been no relevant changes in the ref being pushed. And even when there are changes between origin/main and the ref to be pushed, we'd like to limit the validations to only those changes, if possible. So, we need some infrastructure for identifying the changed files.

Lastly, we'd like to keep the user informed about what checks are being run and print out information about what state we've left the tree in. If we can auto-correct some of the issues, that would be a nice thing to do, too.

The script below will run ktfmt over a Kotlin codebase and fail the push if any of the files are not well-formatted. It does also use a custom Gradle plugin we built around ktfmt, which accepts a --run-over change set. This script will leave the fixed formatting changes in the tree, awaiting the developer to intervene, commit, and try again.

The script below can be trivially adapted to any other number of tools, and you can include a few such scripts in your ./scripts/pre-push.d. For example, my current codebase has one for ktfmt, and a very similar one for Square's Gradle Dependencies Sorter.



#!/bin/bash
#
# This script looks for Kotlin files that have been changed locally and
# ensures that they are correctly formatted according to ktfmt. This
# script makes an effort to only run on the smallest possible set of
# changed files as opposed to running broadly over the codebase, to
# improve pre-push performance.
#
# The script takes the following steps:
#
#  1. Figure out the name of the remote for your repo on
#     GitHub. If there is no remote (via `git remote`) add one called
#     "your_company"
#  2. Fetch the latest `main` ref from that remote.
#  3. For any commits that are about to be pushed (git push can push
#     multiple references at once), do the following steps, 4-8:
#  4. Find an ancestor commit that is before both origin/main and the
#     commit you're trying to push.
#  5. Compute a list of .kt or .kts files that have changed between that
#     ancestor and the commit being pushed
#  6. Run ./gradlew ktfmtFormatPartial on only those files using --run-over=<list>.
#  7. If there are no formatting changes applied, proceed to push.
#     Otherwise, print a descriptive error message noting that files
#     have been formatted and that the user will need to commit manually
#     and push.

# Print the name of the configured remote (usually "origin")
expected_remote() {
    git remote -v | awk '/git@github.com:Your-org\/your-repo.git \(fetch\)/ { print $1 }'
}

# Returns the name of the your_company origin. If it is not found locally,
# we'll add one called "your_company" (conservative name so it doesn't clash
# with whatever else you have going on.)
ensure_remote_installed() {
    remote_name="$(expected_remote)"
    if [ -z "$remote_name" ]; then
        git remote add your_company "git@github.com:Your-org/your-repo.git"
        echo "your_company"
    else
        echo "$remote_name"
    fi
}

# Fetches, but does not apply, the remote references from GitHub's copy
# of the project.
fetch_remote_refs() {
    remote_name="$1"
    git fetch "$remote_name" main &>/dev/null
}

# Computes a list of the names of the files that have changed between
# two commit hashes.
compute_changed_files() {
    from_hash="$1"
    to_hash="$2"
    git diff --name-only "$from_hash" "$to_hash"
}

# Determines if a file is a kotlin file.
is_kotlin_file() {
    file_path="$1"
    if [[ "$file_path" =~ .kts?$ ]]; then
        echo 0
    else
        echo 1
    fi
}

# Computes which .kts? files have changed.
compute_changed_kotlins() {
    from_hash="$1"
    to_hash="$2"
    changed_kotlins=""
    for changed_file in $(compute_changed_files "$from_hash" "$to_hash"); do
        if [ "$(is_kotlin_file $changed_file)" -eq 0 ]; then
            changed_kotlins="$changed_file $changed_kotlins"
        fi
    done
    echo "$changed_kotlins"
}

# This finds a common ancestor between origin/main and whatever commit
# is being pushed. The idea here is that the local tree may not be
# rebased onto origin/main itself, so we need to look backwards in
# origin/main to find a commit that *is* in our history. This is the
# developer's current marker for origin/main.
compute_from_hash() {
    remote_name="$1"
    local_hash="$2"
    git merge-base "$remote_name/main" "$local_hash"
}

# Given two lists of strings compute the insersection of the two lists,
# e.g., if A="foo bar", and B="foo", the intersection is "foo".
compute_intersection() {
    intersection=""
    foo="$1"
    bar="$2"

    for f in $foo; do
        for b in $bar; do
            if [[ "$b" == "$f" ]]; then
                intersection="$intersection $b"
            fi
        done
    done

    echo $intersection | sort | uniq | xargs
}

# Checks the result of the ktfmt task to see if formatted any files.  If
# files were formatted, fail the hook and emit an error. If none were
# formatted, continue to exit the hook successfully.
fail_if_any_formatted() {
    changed_since_main="$1"
    changed_after_fmt="$(git diff --name-only | xargs)"
    changed_in_both=$(compute_intersection "$changed_since_main" "$changed_after_fmt")
    if [ ! -z "$changed_in_both" ]; then
cat <<- EOF
The following files were not formatted correctly and have been fixed locally. Please commit them and try your push again.
$changed_in_both
EOF
        exit 1
    fi
}

# Runs ktfmt over any changed .kts? files.
validate_ktfmt() {
    remote_name="$1"
    to_hash="$2"
    from_hash="$(compute_from_hash $remote_name $to_hash)"

    changed_kotlins="$(compute_changed_kotlins $from_hash $to_hash | xargs)"
    if [ -z "$changed_kotlins" ]; then
        exit 0
    fi

    echo "Running ktfmtFormatPartial over changed Kotlin files: $changed_kotlins ..."
    ./gradlew ktfmtFormatPartial --run-over="$changed_kotlins" &>/dev/null
    fail_if_any_formatted "$changed_kotlins"
}

remote_name=$(ensure_remote_installed)
fetch_remote_refs "$remote_name"

while read localname localhash remotename remotehash; do
    validate_ktfmt "$remote_name" "$localhash"
done

exit 0

In my opinion, the most interesting part of this script is really the git merge-base call, used to compute a common ancestor between origin/main and the current ref. This is useful since you may not be rebased onto the remote's main when the hook runs. But if origin/main is always correct, then the diff that is presented between origin/main and your ref will encapsulate the entirety of your responsibility.

Conclusion

Well, there you have it. After writing a few of these, I put my "best of" playlist up for all to see. Each of these scripts can be found in this Gist. Let me know if you have a better way of doing this, or any ideas for improvements!

Should You Adopt a Cross-Platform Client Framework?

Jameson — Wed, 13 Dec 2023 02:07:25 +0000

Thesis: the decision to use a cross-platform framework is mostly a business one, and somewhat a technical one.

What Problem Is Being Solved?

Recently I've been building spreadsheets with the goal of comparing different aspects of iOS, Android, and the web. For example, here's a high-level comparison of the basic modern tools, languages, frameworks, and architectures employed on each:

Aspect	Android Native	iOS Native	ReactJS
IDE (Development Environment)	Android Studio	XCode	VS Code
Project Creation Template	Android Studio templates	Xcode templates	npx create-vite
Language	Kotlin	Swift	TypeScript
UI Framework	Jetpack	SwiftUI	ReactJS
Test Framework	JUnit	XCTest	Jest
REST/Networking Tool	Retrofit	URLSession, Alamofire	Axios, fetch
Persistence/Database Library	Room	SwiftData
Lint Tool	Android Lint, KTlint	SwiftLint	ESLint
Reference Project on GitHub	Now in Android		Bulletproof React
DI Tool/Framework	Dagger	Needle	React Context
Build Tool	Gradle	Xcode Build System	NPM
Code Minifier	R8	ld	Terser
Asset Bundler	AAPT	actool, momc, ibtool	Webpack
Async Image Loading Tool	AsyncImage (Coil)	AsyncImage (SwiftUI)	Browser
Concurrency Framework	Coroutines	async/await (Swift)	async/await (JavaScript)
Reactive Data Flow Framework	Kotlin Flow	Combine	RxJS
Common UI Architecture	MVVM	MVVM	Components

In short, that's a lot for any one person to become an expert in. Most of us struggle just to develop expertise in a single one of these platforms. So, as an engineering manager, you have essentially only three solutions to this problem.

Hire specialists who can handle each platform
Look for opportunities to "write once, run everywhere," so that you can do more with less
Hire extremely senior people who have experience in all three platforms

It doesn't take a spreadsheet to figure out that (1) and (3) are going to be more expensive in the short term: hiring more people--or more people who are also quite senior--is going to cost. As a result, (2) looks pretty attractive.

How Tech Businesses Evolve

We need to also discuss, for a moment, the natural evolution of a tech startup.

Every business needs a website, and most businesses start with a web app. Websites run on every device, so you can get by with this for a bit, while you're proving viability (and thus, the ability to hire).

In the US, once the business looks viable, it makes sense to build an iOS app. The US' Disposable Income class is mostly using iOS, and these are the folks you need to target to make money. Independent of any interesting technology choices, apps can attain significantly higher engagement than the web. For one, the continued presence of your little launcher icon on an end user's device is free marketing. And more importantly, Push Notifications are the modern replacement of web marketing emails. Even many established apps continue to send notifications that are barely better than this:

"Have you thought of our product today? Open the app to engage and monetize."

The last client that US businesses build, when they've entered the growth/scale phase, is an Android app. As noted above, the Disposable Income class is mostly using iOS. And, it's also true that there are slightly fewer Android users in the US, anyway. All that aside, it is nonetheless true that you can greatly expand your addressable mobile market in the US by building an Android app. So US companies do. More launcher icons, more push notifications, more moolah.

Mid-late growth phase businesses are Android businesses. Outside of the US, UK, Canada, and Australia, everyone uses an Android phone. If you want to make money in Europe, Asia, Latin America, or Africa, you are in the Android business.

These phases of evolution are important to keep in mind because technical decisions (and hiring) in an engineering department often track these phases. Explicitly: you will likely make different choices about client platforms depending on which stage your business is in.

What Are the Options?

At a high level, you can either use cross-platform frameworks or not. Let's talk about the options that exist within the "okay, let's use them" category. There are three cross-platform frameworks worth considering right now: React Native, Flutter, and Kotlin Multiplatform. We can compare these along similar vectors to what we did above:

Aspect	KMP	Flutter	React Native
IDE (Development Environment)	IntelliJ IDEA	VS Code or IntelliJ IDEA	VS Code
Project Creation Template	KMP Project Generator	`flutter create` command	react-native init
Language	Kotlin	Dart	TypeScript
UI Framework	Jetpack Compose, SwiftUI, React	Flutter	React
Test Framework	Kotlin Test	flutter_test library	Jest
REST/Networking Tool	Ktor	Dio, http	Axios, fetch
Persistence/Database Library	SQLDelight	sqflite	AsyncStorage
Lint Tool	Ktlint	Dart Linter	ESLint
Reference Project on GitHub	People In Space	Flutter Samples
DI Tool/Framework	Koin	get_it	React Context
Build Tool	Gradle	`flutter build` command	Metro
Code Minifier	Platform-specific	Part of `flutter build`	Metro Bundler
Asset Bundler	Platform-specific	Part of `flutter build`	Metro Bundler
Async Image Loading Tool	Platform-specific	Image.network()	Image (React Native)
Concurrency Framework	Coroutines	Dart async/await	JavaScript Promises
Reactive Data Flow Framework	Flow	RxDart	RxJS
Common UI Architecture	MVI	Widgets (MVU)	Components (MVU)

React Native

Since every business starts with a stock of web developers, React Native is an attractive option since it's so similar to React.js. Your web engineers have to reach outside their comfort area a little bit, but can essentially be redeployed to mobile with a similar skill set. If your startup is not able to hire iOS specialists in the near term, this is a pretty good pathway to get those Push Notification bucks.

But React Native can be slow. And if your app needs any native iOS functionality, the bridging code can get unruly. It's also true that it's easier to hire native iOS engineers than it is to hire a web engineer and then train them to work on an iPhone. The closer your stay to the "default" platform recommended by Apple, the easier it will be to hire for iOS in the future, too. In short: if you have funding to scale your Engineering team over the next 6-12 months, and know that you'll be able to staff iOS engineers, don't mess around with RN.

Flutter

Flutter's Dart language gets compiled directly down to ARM machine code, which avoids one of React Native's biggest gripes. However, whoever is writing this app will need to learn Dart. This either means hiring specialists (we were trying to avoid this, right?) or having your web/iOS folks learn Dart. Fortunately, Dart is kind of like if TypeScript and Java had a bastard child. So, it's fairly accessible to a broader technical audience.

Like React Native, Flutter has a pretty good story around web interop. For these reasons, it wouldn't be outrageous to start a company with Flutter. You could have web, iOS-- and Android for free. However, most companies don't start with Flutter, they start with React.js on the web. Adopting Flutter once you already have a large React.js investment makes less sense: you'd probably either pick React Native for its similarity or just create a discrete iOS codebase.

Kotlin Multiplatform

Let me start by stating full-throatedly that Kotlin is the best front-end language. Line break.

I am saying this as an Android engineer, yes, but StackOverflow agrees with me. TypeScript is a nice improvement to JavaScript, but it is still a weird bolt-on language. Swift is 5,000 times better than Objective-C and is almost as good as Kotlin. But, Kotlin is a little newer. It had the benefit of learning from these languages, iterating on their weaknesses, and yielding a more polished result.

At the moment, I don't think it makes any sense to build web products with KMP, although it may in the future. There are really two camps that would think seriously about adopting KMM in my view:

International startups that built an Android app first, and are now expanding to iOS
American startups that have a web app, and are at their "how do we do mobile?" inflection point.

KMM makes a lot of sense outside the US, where the mobile economics are different. If you start with a modern Kotlin Android app, building a SwiftUI shim in KMM would be an awesome way to expand your userbase -- while not investing fully.

For the American web startup, KMM could still be attractive. KMM works in a progressive way, where you get to decide how much of a specific platform is implemented natively or not. So, you could decide "okay, we've got our React.js web app, but we're going to share some code across iOS and Android in this new codebase." Business logic could be written in Kotlin (a beautiful language), while large portions of the apps are built in Compose and Swift/SwiftUI. This allows you to stay somewhat true to the platforms, making long-term hiring for mobile specialists easier, too. (Dart and RN don't have this benefit.)

Shared Architecture, Separate Platforms

The last option I want to talk about-- and indeed the one that I think is right for many monetized companies-- is to maintain three separate frontend codebases that share architectural similarities.

When iOS, Android, and the Web first emerged, they were very different from one another. JavaScript is not much like Objective-C. And syntax aside, Java is not much like either of those two. jQuery isn't like UIKit, which isn't like writing XML for Android Views.

But that isn't the case anymore. React.js, SwiftUI, and Jetpack Compose all agree on composable hierarchies of UI components using uni-directional dataflow. TypeScript, Swift, and Kotlin all share some broadly similar syntax and language constructs. So, a valid question to ask is this:

Can we exploit the similarities in the three platforms to keep the codebases ideologically consistent?

There are some clear benefits to doing this. Firstly, it'll be easy to hire platform specialists in perpetuity, since you're not "doing anything weird" in your tech stack. Secondly, you'll have no performance hits, since everything is native. And thirdly, there'll be no added complexity on any individual platform when bridging from the cross-platform framework into the native code.

In practice, there are unique technical challenges to either approach - aligning three platforms through organizational cohesion or aligning three platforms through cross-platform frameworks. Cross-functional collaboration with your UX and Product designers becomes increasingly important when you operate in multiple codebases: the Engineering team needs to be using the same component names as the UX team, to stay aligned.

When to Choose Each

"It depends," is usually a copout of an answer. So I'm going to make some opinionated suggestions.

If you're starting from scratch and need to prove viability, either start with a React web app or start with Flutter, which are acceptable web solutions.

Next, if you didn't pick Flutter and are figuring out how to expand into mobile, you should pick:

React Native, if you don't have a line of sight to fund specialists on your Engineering team over the next year.
KMM, if you are in a market where Android dominates, and will enter the market on Android first.
A native iOS app, if you are American, and will enter the market on iOS first.

More nuanced situations arise for SMBs, who already have some combination of the technologies above. But, generally speaking, if you are growing as a business and Engineering team, you should be migrating away from cross-platform frameworks into discrete native codebases that share similar architecture and concepts.

If you are a revenue-stable US SMB, and iOS is an important portfolio driver, your best investments are probably in native iOS or React Native, depending on the context of your web staff and product. If you are a revenue-stable SMB outside the US, your best investment is probably KMM.

There are yet other businesses that support more client platforms: for example, Netflix, Crunchyroll, etc. need to support a half-dozen-or-so TVs and Gaming platforms. To the extent that these companies can envelope a functional web app into some kind of wrapper while maintaining acceptable media performance, that option will almost certainly be a good trade of engineering effort and complexity vs. end-user monetization per platform.

Wait, What About Tech Factors?

At the beginning of this article, I argued that multi-platform is a business decision. And, my argument has essentially been that you need to look at monetization of iOS, Web, Android, and compare it to your Engineering team's budget. If you have a lot of money, build native and don't look back. If you don't, you can compromise and optimize for the platforms that are most important to driving revenue in your business.

But, technology can be an important aspect of "what drives revenue for your business?" If you intend to have a media-rich application with smooth animations, real-time widgets, and a high frame rate, you may not be able to achieve your business goals with some of these choices.

There are many high-profile deep dives around mobile teams entering and exiting the React Native arena - here are two well-known organizations that have come or gone:

Anecdotally, Shopify shed some significant native mobile headcount through regrettable attrition when they adopted RN at their established business.

For Flutter and KMP, it may be too early to have "here's our decision after a few years"-style articles from high-profile businesses. But, they also have their own naysayers:

While it is true that all of RN, Flutter, and KMP support bridging into native code, this type of code tends to invalidate the productivity benefits of adopting a cross-platform framework to begin with. Not only are folks writing native code anyway, but they also have the added complexity of fighting an additional framework, too.

Cross-platform is always a financial or organizational shortcut, inherently more cumbersome than native development. And while there is some developer productivity to be gained in code reuse, the gains may not be linear with the size of an app or its Engineering team. On the other hand, shortcuts are sometimes necessary in business for survival. If your options are to ship your web app on iOS via RN, or to lose market opportunities on iOS for lack of staffing/expertise: that's an obvious choice.

Well, that's it. Would love to hear your thoughts!

Modern Async Primitives on iOS, Android, and the Web

Jameson — Wed, 06 Dec 2023 02:34:06 +0000

Let's Talk About Asynchronicity

When you began your programming journey, you started with synchronous, serialized code. One operation follows another, and so on, along a single timeline.

Soon after, you likely discovered the ability to execute code along multiple timelines. With this capability, your code might still be synchronous, even if it splits its tasks across two timelines. But, your concurrent code could also be asynchronous, meaning that your main timeline continues along while some results are deferred until later.

(Thanks @AlisdairBroshar for the image.)

Arguably, a more interesting version of asynchronicity exists when multiple tasks are scheduled along a single timeline. This is intuitively easy to understand: you go about your day on a single timeline, and you probably juggle many unrelated tasks as you do so. You have to suspend one of your tasks to focus on the next one, and then the next one, etc. But unfortunately, those suspended tasks don't make any progress while you're not working on them.

(Image credit to Sonic Wang @ DoorDash Engineering)

In this space, we also have the somewhat related term blocking. Java's NIO library is one well-known non-blocking tool used for managing multiple tasks on a single Java thread. When listening to sockets, most of the time a thread is just blocked, doing nothing until it receives some data. So, it's efficient to use a single thread for monitoring many sockets, to increase the likelihood of the thread having some actual work to do. The Selector API does this but is notoriously challenging to program well. Instead, developers use frameworks like Netty which abstract some of NIO's complexity and layer on some best practices.

(Thanks GeeksForGeeks.org for the image.)

In the current generation of programming languages, there has been a renewed effort to simplify asynchronicity through structured concurrency. Language facilities like Swift's async/await or Kotlin's Coroutines allow for a unit of work to be suspended and resumed within the context of one or more timelines. Below, we're going to explore how some of these tools work.

Swift: async/await & AsyncSequence

Swift's structured concurrency support was announced at Apple's WWDC '21 conference. Kavon @ Apple gives a great intro to the topic here. Swift's async/await support allows you to yield a flow of execution in your program while awaiting the result of a task. One example given in Kavon's presentation is to download some data and then await the result:



// Note `async let` here! Two tasks run concurrently.
async let (data, _) = URLSession.shared.data(for: imageReq)
async let (metadata, _) = URLSession.shared.data(for: metadataReq)
// Do other tasks here, while tasks execute
guard
  let size = parseSize(from: try await metadata),
  let image = try await UIImage(data: data)?.byPreparingThumbnail(ofSize: size)
else {
  throw ThumbnailFailedError()
}
return image

The beauty of this code is that all of the async work will be terminated if this block goes out of scope. This simplifies your mental model considerably and avoids entire classes of bugs.

Remark: the data and metadata objects above might look a lot like JavaScript Promises, Rx Singles, or Java Futures, if you've seen any of those before. (We'll talk about Promises, later.)

Swift >=5.5 also supports a new means of unstructured concurrency, where you are responsible for governing the task lifecycle yourself, instead of letting Swift handle it implicitly. For example, you can launch a Task to await the execution of some async function:



Task {
  await someAsyncFunction()
}

When using the Task construct, you need to be careful to cancel() the Task at the appropriate time when it goes out of scope.

Beyond these single-result-oriented constructs, Swift >=5.5 also supports async operations over collections. AsyncSequence, also announced at WWDC21 builds on earlier streamed value solutions like RxSwift's Observable or Combine's PassthroughSubject. But unlike those constructs, AsyncSequence can yield execution between iterations.

Let's see in action. First, let's take some ordinary for loops and wrap them in Tasks. As noted above, the Task object will provide an execution context on the main actor, in which async work could occur.



Task { @MainActor in
    for f in [1, 2, 3] { print(f) }
}
Task { @MainActor in
    for s in [10, 20, 30] { print(s) }
}

As you would expect, since there isn't any use of async anywhere, these just run sequentially. The output is:

Now, instead, let's try changing the array to an AsyncSequence, and running it again. Importantly, everything will still be on the main actor.

Note for the code below: .publisher transforms the array into a Combine publisher, and .values creates an AsyncSquence from that Publisher.



Task { @MainActor in
    for await f in [1, 2, 3].publisher.values { print(f) }
}
Task { @MainActor in
    for await s in [10, 20, 30].publisher.values { print(s) }
}

This time, the loops yield execution after each value is emitted. The two tasks end up interleaving their output:

The two tasks collaborate on a single thread of execution, almost as if they were operating concurrently. Pretty neat, right?

One convenient place an AsyncSequence shows up is in this short-hand to read a network request's data, line-by-line. This can be achieved by async-iterating over the lines property on URL:



let rickMortyApi = "https://rickandmortyapi.com/api"
if let data = URL(string: rickMortyApi) {
    for try await line in data.lines {
        print(line)
    }
}

Another place you might encounter AsyncSequence is in SwiftUI's property wrappers. For example, let's suppose you have a field @Published var credentials: [Credential] in some @ObservableObject:



class User: ObservableObject {
  @Published var id: String
  @Published var credentials: [Credential]
}

You could iterate over this value like so:



for await credential in $user.credentials {
  // use credential
}

Side note, if you Migrate from the Observable Object protocol to the Observable macro, a lot of your $ binding calls will probably go away, rendering this kind of iteration unnecessary.

This new functionality is not without some open questions, though. In his excellent blog from April 2022, Using new Swift Async Algorithms package to close the gap on Combine, John O'Reilley notes:

As developers have started adopting the new Swift Concurrency functionality introduced in Swift 5.5, a key area of interest has been around how this works with the Combine framework and how much of existing Combine-based functionality can be replaced with async/await, AsyncSequence etc. based code.

In that blog, John makes a note of the AsyncExtensions library, which has several cool Features like merge(), zip(), etc., that you'd expect from other stream/collections API surfaces.

You might also prefer to look at Apple's own evolution library for AsyncSequence, which offers a mostly-similar feature-set, called swift-async-algorithms. This library is more likely to be a conservative staging ground for forthcoming Swift language APIs.

Kotlin: Coroutines, Flow

Many ecosystems have adopted the async/await style of structured concurrency, but Kotlin took a slightly different approach. Coroutines is an older technology: the first usage of the term dates back to the late fifties and early sixties. The Kotlin implementation provides several improvements over prior implementations, however, such as native language integration, readability, structured concurrency, and error handling.

While many languages use async to denote an asynchronous function, Kotlin instead marks functions with suspend. Similar to what we saw with AsyncSequence, suspend introduces a suspension point where execution may be yielded.



suspend fun getNetworkData(): Result<Response, Error> {
  // ... well, how bout it?
}

Just like Swift has support for structured concurrency, Kotlin can nest coroutines, too, s.t. canceling a top-level coroutine will also cancel its child coroutines. Let's consider an example. Below we use the launch coroutine builder to initiate a parent coroutine, and another launch coroutine builder to create a child coroutine within it.



coroutineScope.launch { // When I am canceled,
  launch { // I will cancel, too.
    getNetworkData()
  }
}

In Kotlin, the closest thing to Swift's Task is GlobalScope.launch, which can be used to initiate a coroutine that is not bound to its parent's scope. Use of GlobalScope is generally not recommended, since it will require you to carefully manage the lifecycle of the returned Job object, calling cancel() when appropriate.

Consider the bit of code below as an example, wherein a coroutine is launched within another coroutine:



var inside: Job? = null
GlobalScope.launch {
  inside = GlobalScope.launch {
      delay(500)
      print("An inside job!")
  }
}.apply {
    delay(100)
    cancel()
}
inside!!.join()

This code will print:



An inside job!

If you remove the GlobalScope designation, then the code prints nothing, since inside is automatically canceled.

Kotlin also has a construct for asynchronous collections/streams. Kotlin's version of AsyncSequence is called a Flow. Just as Swift's AsyncSequence builds upon prior experience with RxSwift and Combine, Kotlin's Flow APIs build upon earlier stream/collection APIs in the JVM ecosystem: Java's RxJava, Java8 Streams, Project Reactor, and Scala's Akka.

Let's see Flow in action:



runBlocking {
    launch {
        flowOf(1, 2, 3).collect {
            yield()
            println(it)
        }
    }
    launch {
         flowOf(10, 20, 30).collect {
            yield()
            println(it)
        }
    }
}

The code above is single-threaded. If we weren't using Flow, we might expected the values to be emitted serially: 1,2,3,10,20,30. But, since each coroutine yields execution upon receiving a value, the outputs interleave:

The code above uses an explicit yield() operator to show where the code yields execution. As a technical note, collect { } itself does not suspend, without yield(), even if the Flow would otherwise.

In practice, you find Flows used heavily between the View and View Model layers of most modern apps using MVVM/MVI design patterns. Views commonly collect state updates from View Models using Flows.

JavaScript: aysnc/await, Promises, AsyncIterator/AsyncGenerator

async/await support has existed in Node.js and web browsers since 2017. Famously, await can be used to wait for a Promise to resolve or reject:



async function fetchData() {
    const fetchPromise = fetch("https://rickandmortyapi.com/api")
    console.log("Awaiting your data...")
    const data = await (await fetchPromise).json()
    console.log("The data has arrived: ", data);
}

(async () => await fetchData())()

The code above works similarly to the unstructured concurrency primitives we saw with Swift's Task and Kotlin's GlobalScope.launch. The fetch function initiates some asynchronous work, bundles it up into a Promise object, and yields execution so the lines below it can proceed. Finally, we await the result of the Promise, and extract JSON data received from the network.

Unlike Swift and Kotlin, JavaScript does not have any native support for structured concurrency. In fact, it doesn't even have native support for canceling unstructured tasks. There could be various reasons for this: JavaScript was one of the first mainstream languages to adopt async/await and its interpreters have traditionally been single-threaded. Nevertheless, this oversight has compelled the ecosystem to innovate its own unique solutions. As a well-known example, Axios provides a CancelToken to facilitate task cancellation within that library.

However, JavaScript does have async collection primitives analogous to those provided by Swift and Kotlin: AsyncIterator and AsyncGenerator.

Much like we saw with Swift's AsyncSequence, adding the await keyword into the generic for loop will create a suspension point wherein the JavaScript interpreter can yield its execution to other tasks. (See for await...of documentation from Mozilla.)



async function* asyncGenerator(multiplier: number = 1) {
  yield 1 * multiplier;
  yield 2 * multiplier;
  yield 3 * multiplier;
}

(async () => {
  for await (let i of asyncGenerator(0)) console.log(i);
})();

(async () => {
  for await (let i of asyncGenerator(10)) console.log(i);
})();

This code is single-threaded, but we again see two tasks using cooperative multitasking to yield execution to one another. The result is familiar:

Wrapping Up

Well, there you have it: a broad overview of structured concurrency and asynchronous collections programming in Swift, Kotlin, and JavaScript.

To help illustrate, here's a table comparing some of the machinery we discussed, across each of the three languages.

	Swift	Kotlin	JavaScript
Launch async work (structured)	`async let`	`launch { }`	N/A
Launch async work (unstructured)	`Task { }`	`GlobalScope.launch { }`	`Promise()`
Cancel async work (unstructured)	`Task.cancel()`	`Job.cancel()`	N/A
Await completion of work	`await`	(Implicit)	`await`
Mark function as asynchronous	`async`	`suspend`	`async`
Async collection	`AsyncSequence`	`Flow`	`AsyncIterator` / `AsyncGenerator`
Async iteration syntax	`for await x in seq`	`flow.collect { }`	`for await (let x of gen)`

Splitting SwiftData and SwiftUI via MVVM

Jameson — Tue, 31 Oct 2023 22:20:54 +0000

SwiftData was announced at WWDC in June 2023. Its primary goal is to enable data persistence in a SwiftUI app with as little fuss as possible.

Most of the tutorials and snippets we see on the web show it being used directly in a SwiftUI View. For example, let's consider the minimal code to query, insert, and delete items from a list.

Suppose we have a model of an item that we'll want to display in a list. For example, here's the @Model that XCode generates if we select "use SwiftData" in the project template:



@Model
final class Item {
    var timestamp: Date

    init(timestamp: Date) {
        self.timestamp = timestamp
    }
}

With very little code, we can use the @Query property wrapper to read some existing items and show them in a list:



struct ContentView: View {
    @Query private var items: [Item]

    var body: some View {
        List {
            ForEach(items) { item in
                Text(item.timestamp, format: Date.FormatStyle(date: .numeric, time: .standard))
            }
        }
    }
}

Of course, we'll also need some way to add and remove those items. For whatever reason, those actions happen through a separate ModelContext object:



struct ContentView: View {
    @Environment(\.modelContext) private var modelContext
    @Query private var items: [Item]

    var body: some View {
        VStack {
            Button(action: {
                modelContext.insert(Item(timestamp: Date()))
            }) {
                Label("Add Item", systemImage: "plus")
            }
            List {
                ForEach(items) { item in
                    Text(item.timestamp, format: Date.FormatStyle(date: .numeric, time: .standard))
                }
                .onDelete(perform: { offsets in
                    for index in offsets {
                        modelContext.delete(items[index])
                    }
                })
            }
        }
    }
}

Above, we've included a simple button to add new Items, and a swipe-to-delete interaction that can delete Items. Here's what the UI ends up looking like:

The magic here, of course, is that the items will persist even after the app is killed: that's the point of using SwiftData.

First Attempt at MVVM & SwiftData

Let's now suppose that we want to split apart our code and introduce an MVVM pattern. Doing so will help us to create testable, reusable layers in our app.

We have a few new challenges here, using SwiftData. Firstly, we'll need a way to get access to the ModelContext outside of the SwiftUI View. Let's try a naive approach of using the same @Environment property binding to access ModelContext from our new ViewModel, instead of the ContentView. We might end up with something like below. Note that all data-management actions have been moved out of the ContentView and now live in a new class ViewModel.



struct ContentView: View {
    @State private var viewModel = ViewModel()

    var body: some View {
        VStack {
            Button(action: {
                viewModel.appendItem()
            }) {
                Label("Add Item", systemImage: "plus")
            }
            List {
                ForEach(viewModel.items) { item in
                    Text(item.timestamp, format: Date.FormatStyle(date: .numeric, time: .standard))
                }
                .onDelete(perform: { offsets in
                    for index in offsets {
                        viewModel.removeItem(index)
                    }
                })
            }
        }
    }
}

And a really simple-minded ViewModel to go with it:



@Observable
class ViewModel {
    @ObservationIgnored
    @Environment(\.modelContext) private var modelContext

    var items: [Item] = []

    func appendItem() {
        modelContext.insert(Item(timestamp: Date()))
    }

    func fetchItems() {
        do {
            items = try modelContext.fetch(FetchDescriptor<Item>())
        } catch {
            fatalError(error.localizedDescription)
        }
    }

    func removeItem(_ index: Int) {
        modelContext.delete(items[index])
    }
}

At this point, we try to run our simple app and we get this error:

Accessing Environment<ModelContext>'s value outside of being installed on a View. This will always read the default value and will not update.

Darn.

Updating App integration

Let's go back and look at our App class for a moment. The simplest integration for SwiftData is to inject the ModelContext into the WindowGroup, like this, which makes it available to SwiftUI:



@main
struct DataPlaygroundApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
        .modelContainer(for: Item.self)
    }
}

It's worth noting that mainContext is a property inside of a ModelContainer. So, instead of accessing the ModelContext via @Environment, maybe our ViewModel could access it through the ModelContainer.

Let's remove that modelContainer(...) line in the App altogether and try a different approach. Instead of accessing it from the View layer at all, let's create a data source class.

Our App now looks like this:



@main
struct DataPlaygroundApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

Defining a Data Source

Now, let's define an ItemDataSource that will own the ModelContainer:



final class ItemDataSource {
    private let modelContainer: ModelContainer
    private let modelContext: ModelContext

    @MainActor
    static let shared = ItemDataSource()

    @MainActor
    private init() {
        self.modelContainer = try! ModelContainer(for: Item.self)
        self.modelContext = modelContainer.mainContext
    }

    func appendItem(item: Item) {
        modelContext.insert(item)
        do {
            try modelContext.save()
        } catch {
            fatalError(error.localizedDescription)
        }
    }

    func fetchItems() -> [Item] {
        do {
            return try modelContext.fetch(FetchDescriptor<Item>())
        } catch {
            fatalError(error.localizedDescription)
        }
    }

    func removeItem(_ item: Item) {
        modelContext.delete(item)
    }
}

A few things to notice here. The class uses @MainActor, since we must access mainContext from the main actor. If you don't include @MainActor, you'll get a compile error like this:

Main actor-isolated property 'mainContext' can not be referenced from a non-isolated context

Also, I've made the component a singleton so we only have one instance of the ModelContainer. (There are many other ways to achieve this, this was easy for demonstration purposes.)

Now, our simple ViewModel just calls out to the data source. When items changes, since the ViewModel is @Observable, the ContentView will automatically respond to it and render the new items.



@Observable
class ViewModel {
    @ObservationIgnored
    private let dataSource: ItemDataSource

    var items: [Item] = []

    init(dataSource: ItemDataSource = ItemDataSource.shared) {
        self.dataSource = dataSource
        items = dataSource.fetchItems()
    }

    func appendItem() {
        dataSource.appendItem(item: Item(timestamp: Date()))
    }

    func removeItem(_ index: Int) {
        dataSource.removeItem(items[index])
    }
}

And there you have it! You can re-use the ItemDataSource in any number of your SwiftUI View/ViewModels, and you can start building up tests around each component in isolation.

Let me know in the comments what you think of this approach.

Simple Runtime Feature Gating on Android

Jameson — Sat, 29 Oct 2022 11:21:59 +0000

I recently saw a tweet that got me thinking:

// Detect dark theme var iframe = document.getElementById('tweet-1586230874915409921-312'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1586230874915409921&theme=dark" }

The key point here is absolutely true. On the backend or frontend, you can deploy code whenever you want, all day every day. Your users will instantly get these updates.

That's not how it works with mobile apps. If you want to roll an updated version of your app, you'll at minimum need to upload the new version to Google or Apple, and await their review/acceptance.

If you work on a mobile app team, you're probably doing a weekly or maybe bi-weekly release. In this environment, it'll take even longer before you can get a fix out.

But even if you could update your app every time you make a change - should you? No. Each update you make is potentially an annoyance for the mobile app user, and a waste of their bandwidth if you're updating too frequently.

"Rollback" refers to pulling a defective deployment and going back to the last known working version. In the Google Play and Apple App stores, there is nothing exactly like this. So, the common approach to rollback involves re-publishing the content of an old release, but with a new, later version number. It is in fact a new release artifact, it just functions like an older one.

A better solution is runtime feature-gating. The idea of feature-gating is basically to put every big new change behind a toggle that you can control remotely from a server under your control. Whenever you make a change in your mobile app code, you can add a flag for it, which controls whether or not to use the new code.

Big companies generally have robust tooling for this - and it may be tied into a larger experimentation and analytics framework. But the basic concept is very easy to apply in your own DIY app; you don't need a lot of complexity to get most of the value. Let's look at how.

Let's suppose we've built a Halloween-themed screen for our app, and want to show it only on October 31st. There are a few ways we could achieve this:

Hard code a date check on the client;
Try to update the Play Store before and after halloween with new app versions;
Use a runtime feature gate.

Option 1 is alright, but if something goes wrong - faulty date logic - we'll be celebrating Halloween longer than we wanted.

Option 2 assumes that our timing with the Play Store will work out perfectly, and also creates a lot of operational churn.

Option 3 looks really appealing. When Halloween comes, all we have to do is update a file on our backend (or saved to any public location like GitHub Pages, even.) If something goes wrong, we just update the file and no one knows any wiser. Either way, we can update the file at the end of the holiday.

To gate this change on the client with a runtime feature gate, let's take the following approach:

Create a flat JSON file and make it available for download somewhere;
When the client starts up, go download this file and check its contents;
Check the feature enablement state before showing the user the Halloween-specific feature(s).

A simple incantation of this JSON file would look like this:

{
  "features": [
    {
      "name": "halloween_screen"
      "enabled": true
    }
  ]
}

Now, on the client, let's whip together a little network client with Retrofit and KotlinX Serialization. This code will download this file and parse it into a data type.

interface RuntimeFeaturesService {
  @GET("features.json")
  suspend fun features(): Features

  companion object {
    fun instance(): RuntimeFeaturesService {
      return Retrofit.Builder()
        .baseUrl("https://raw.github.com/jameson/proj/")
        .addConverterFactory(
          Json.asConverterFactory(
            MediaType.get("application/json")
          )
        )
        .build()
        .create(RuntimeFeaturesService::class.java)
    }
  }

  @Serializable 
  data class FeatureList(features: List<Feature>) {
    @Serializable 
    data class Feature(name: String, enabled: Boolean)
  }
}

Of course, you'll probably want to layer on some caching and a repository on top of this. But you can start using the new check immediately:

val halloweenFeaturesEnabled = 
  runtimeFeaturesService.features()
    .filter { f -> f.name == "halloween_screen" }
    .firstOrNull()
    ?.enabled ?: false

if (halloweenFeaturesEnabled) {
  binding.pumpkinImage.visibility = View.VISIBLE
} else {
  binding.pumpkinImage.visibility = View.GONE
}

When it's time to get rid of the pumpkin image, just update the JSON:

{
  "features": [
    {
      "name": "halloween_screen"
      "enabled": false
    }
  ]
}

The example above is intentionally simplistic, and uses minimal third-party tooling to achieve its goals. You may quickly want to pivot towards a more robust solution like Firebase Remote Config.

Fakes & Mocks on Android: Well Partner, that depends.

Jameson — Sat, 01 Oct 2022 18:56:26 +0000

Over the course of 2021 and 2022, software testing fashion has swayed away from mocks, towards fakes. So when should you use one or the other? As with most serious engineering questions, the answer is "it depends."

Yet, in recent times mocks have reached levels of hate not seen since Disco Demolition Night, when America gave up on disco and burned all of their records.

// Detect dark theme var iframe = document.getElementById('tweet-1576231597304516608-906'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1576231597304516608&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1507416343796191236-550'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1507416343796191236&theme=dark" }

Even our cultural guru and spiritual leader, the fabled Jake Wharton (🙌), shares this world view:

// Detect dark theme var iframe = document.getElementById('tweet-1104012876048752640-705'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1104012876048752640&theme=dark" }

A newer piece of Android documentation now stops short of wholly recommending fakes, but does only showcase how fakes work - not mocks.

An Appeal to the Classics

The canonical essay on test doubles is Martin Fowler's famous 2007 essay, Mocks aren't Stubs. If you haven't ever read it, close this tab, and go read that instead. You'll learn more, and I won't be offended. It offers now-standard definitions for the different types of test doubles, including fakes and mocks:

Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).

Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

One of my favorite parts of this article is where Martin explores the two schools of thought, purists and mockists - or the "Detroit" style vs. the "London" style. This is the most correct framing of the debate, as one between schools of thought.

In full disclosure, I am a purist of the Detroit school. I don't care what behaviors a component need exercise; I prefer to look at its results. Even so, I will continue arguing a place for mocks, ✊.

How Sociological Trends Work

Before we get into the merits, I'd like to point out that we are indeed talking about a sociological fad more so than a technical evolution. I'm not pointing this out to be contrarian or diminutive, rather so that we can move forward with appropriate and metered expectations.

The Gartner Hype Cycle describes how societies respond to new technologies. New tech comes in, people don't yet see or understand the rough edges, and so hopes are high. "Wow, this has benefits!" Later, the drawbacks are learned. The cycle ends with a tool that you can use, but "it depends" when.

"Mocks Require Mock-able Code"

One argument I've seen is that you have to write impaired source code to use mocks. Actually, I think the opposite is sort of the case.

In languages that close implementations by default (our beloved Kotlin) you have to create an interface to take advantage of a fake. So, unless you're only doing component-level testing, you'd need to create interfaces for every unit. You'll end up with a lot of silly little interfaces, all in the name of faking.

On the other hand, maybe you should just do component-level testing. Here, I define "Component" as a collection of units (classes, interfaces, data types), that serve some well-bounded purpose. For example, a repository dressed with its production dependencies might be a component. Does every little detail beneath the repository need an interface? No, probably not.

Components are likely the biggest granule that can be reasonably tested on your host's JVM, without needing real implementations on your target device. This is, in my view, the best combination of:

"Right-sized" abstractions with interfaces;
Bang-for-your-buck w.r.t. test coverage;
Most approximating of real world behavior, while still being able to run on your workstation.

But, how would it work with mocks? You don't need to do anything to use mocks, that's arguably part of their charm. They're a great choice for legacy code which has few walls of abstraction. Frameworks like Mockito and MockK will happily spoof final/closed functionalities - even returning null from non-null functions. This is thanks to the Magic of Reflection™. Are all of these good traits? I mean: probably not. However, it is still true that mocks are less impacting to your actual source code.

Reflection is Slow and Spooky

Bars, dude. Bars. This is true.

Even Java, upon introducing the capability to its own language remarked in its documentation:

Reflection is powerful, but should not be used indiscriminately. If it is possible to perform an operation without using reflection, then it is preferable to avoid using it.

Now, it is possible to create a fake that uses reflection (your fake implementation could internally use a dynamic proxy pattern, or something.) And, it is technically possible to create mocks that don't use reflection. You could write a custom, concrete implementation for every single test case - obviously, though, that would be just: wow.

Never-the-less, it is most often the case that mocks = reflection, fakes = compiled contracts. So, fakes do have the usual benefits of a reflection-free existence here: they run faster, and you can catch more bugs in your test implementation through the compiler instead of while running your test.

Fakes Have an Upfront Cost

Fakes may require more up-front investment. Let's suppose you have a happy little interface like this:

Now, you want to use it, say in some use case - maybe you're updating user preferences.



class UpdateUserPreferencesUseCase(
  val userRepository: UserRepository,
  val user: User,
) {
  fun disableAnimations() {
    // ... 
  }
}

And, let's further suppose that your code layout looks something like this:

When you go to write this UpdateUserPreferencesUseCaseTest, you come upon a decision point. Should you create a fake of the UserRepository or should you mock it? And where should that code live, in either case?

For small interfaces, the cost of writing a fake vs. a few mock statements is pretty similar. You can write a fake of two functions very quickly - in about as much time as it takes to write the two mocking clauses.

For larger interfaces, the cost is higher right now. It will take you longer to think through a reasonable fake implementation for 5, 10, 20 function contracts. It will be cheaper to write only the two mock statements your test needs to pass, right now.

So in either case, the fake will take at least as much energy as the mock for right now.

"Fakes Have Lower Maintenance Cost"

Now, the real argument we hear in favor of fakes is that they have lower long-term maintenance cost. But is that true? Long-term maintenance cost also depends on various factors.

Suppose you put your fake in :settings:impl, next to your UpdateUserPreferencesUseCaseTest. At this point, there's no long-term cost. The one fake will have the same maintenance cost as the one file using mocks.

But what happens if you have 5 or more uses of that UserRepository in tests across your code base? Well, the natural evolution would be to end up with 5 different fake implementations, each living nearby to the tests that need them, in different modules. If you end up here, fakes are in fact awful for long-term maintenance.

Of course, there's a better way. A better strategy would have been to put the fake implementation in some re-usable module, say :users:fakes:

If you do this, all of your tests that need a FakeUserRepository can use the one common implementation. As the number of tests that need this component grow, so will you see improved long-term maintenance costs, and amortized returns on your up-front investment to build it. That's essentially Zac's argument in this graph:

But, here's the thing. This has nothing to do with fakes. You can do this with mocks, too! You could create a mocks factory and put it in :users:mocks:



object UserRepositoryMocking {
  fun mock(): UserRepository = mockk {
    // etc.
    every { getUser(any()) } returns User("Joe Swanson")
  }
}

So, it's more how you re-use code that impacts long-term maintenance cost, not something intrinsic about fakes vs. mocks.

If you put your fake implementations next to individual tests, you'll end up having to create multiple fake implementations, which will be costly to maintain.

And vice-versa, if you don't put your mocking code in some central tool, you'll end up with a lot of repeated setup code that you'll have to implement in every test.

Wrapping up

Honestly, one of the things I dislike most about tech articles is when someone just leaves you hanging with "it depends", and thinks they've done something clever with that. Like bruv, we pay you proper quid to get this stuff sorted, innit?

Here are the specific best practices that I think you should bring into your code base.

For interfaces that will see high re-use, write a fake implementation. Place the fake into a re-usable test module so that future tests can leverage it, too.
For larger interfaces, legacy components without interfaces, or components with very few uses ("the long tail"), prefer mocks. This will have lower up-front and lower long-term maintenance cost.
For mocking behavior you use across many tests, create a re-usable factory utility to centralize boilerplate, akin to what you do for fakes.
When authoring new source code, honor the Single Responsibility Principle and Interface Segregation Principle, which will facilitate either of these approaches, as you go forward.

Happy testing 🎉

Bash Completion for Git on Mac OS X Monterey

Jameson — Wed, 19 Jan 2022 07:23:34 +0000

For years now, I've had a working recipe to enable bash autocompletion for git on my Mac. Recently that broke due to a packaging change with Xcode's CommandLineTools. So, let's take a moment to review how this all works, and how we can get it working (again).

Quick background on bashrc files

bashrc files are environment configuration scripts that Bash includes when it starts. You can use them to customize your CLI experience.

However, the single ~/.bashrc file can get unmanageable quickly. So just like any other piece of software, we'd like to template things and break the problem into smaller chunks.

My dotfiles GitHub repo includes the collection of .bashrc template files I use on a day-to-day basis:

jamesonwilliams / dotfiles

Jameson's . (dot) files for Linux

dotfiles

Jameson's . (dot) files for UNIX and Linux family command-line environments.

One goal of these dot files is to converge on a similar experience across some different UNIX/Linux systems that we encounter in the wild:

Ubuntu
Debian
RHEL
Amazon Linux
Mac OS X
etc.

Setup

To install the dot files for a particular user on your system, do:

git clone https://github.com/jamesonwilliams/dotfiles.git
./dotfiles/install.sh

View on GitHub

Basically, they work as follows.

~/.bash_profile is the hook used by Mac OS X to start loading your Bash customizations. Mine does nothing other than call ~/.bashrc.
My ~/.bashrc itself contains no real logic, it just sources a bunch of template files that handle specific sub-problems.
One of these template files is called ~/.bashrc.darwin, and it includes some fixes for Mac-specific quirks. Stuff that's generally useful on Linux or other UNIX flavors doesn't go into this file.

How it used to work on Mac OS X

One of the quirks of using OS X is that tab completion with git doesn't work out of the box. So for years, I've had a little bit of configuration to handle this case:



# In ~/.bashrc.darwin

cli_tools='/Library/Developer/CommandLineTools'
git_core="$cli_tools/usr/share/git-core"
git_completion="$git_core/git-completion.bash"
[ -x "$(which git)" ] && \
    [ -f "$git_completion" ] && \
    source "$git_completion"

If git is installed and the git completion file can be found, we source it into the environment.

How it works now

Recently, that git-completion.bash file disappeared from the CommandLineTools install, and so my tab completion stopped working.

Turns out it now lives here:



/Applications/Xcode.app/Contents/Developer/usr/share/git-core/git-completion.bash

So all that's needed is to update our pathing for git_core in ~/.bashrc.darwin:



# In ~/.bashrc.darwin

xcode_dev_dir='/Applications/Xcode.app/Contents/Developer'
git_core="$xcode_dev_dir/usr/share/git-core"
git_completion="$git_core/git-completion.bash"
[ -x "$(which git)" ] && \
    [ -f "$git_completion" ] && \
    source "$git_completion"

Save the file, reboot your laptop, and profit. Happy git-ing.

Scaling Development of an Android App

Jameson — Tue, 14 Sep 2021 03:31:14 +0000

This article explores different strategies for organizing and packaging Android source code as your organization grows. We also look at ways to divide up ownership across different developers in your organization. We’ll follow a typical growth trajectory of an Android app, starting from a simple toy project and ending up with a large consumer-facing product.

Act 1: The Simple Sample App

When you’re working on a small app by yourself, your project structure likely looks something like a Matryoshka (“Russian-nesting”) doll. At the outermost layer, you’ve got a single Git repo. The repo contains a single root Android Gradle project. In the Gradle project, you’ve got a single Android application module called "app.” Inside the app module, you might have a few small Kotlin packages into which you’ve separated your presentation logic from your domain modeling code. Lastly, each Kotlin package contains a few files/classes.

Right now, you're the only person working on this codebase, and you have access to everything. When it comes time to prepare an apk for installation, there are no intermediate steps. All of the files are compiled and indexed in one big bang.

Act 2: The Prototype That “Made It”

Good news - the business has decided to move forward with additional investment in your little Android app. As you build out more features, you’re going to add more Kotlin files and more Kotlin packages. Separating your code into small files and packages will encourage clean single-responsibility components, and self-document the relationships between them. At this phase, your project might have a hierarchy of Kotlin packages and dozens of Kotlin files/classes.

Act 3: A Real Production App

After a while of building features for the business, you’ve got a "real" Android app on your hands. A few key changes occur at this phase.

Firstly, you might have additional developers working with you on the same codebase, now. And for now, that isn’t too much of a problem. There are only 2-5 of you, and you’re staying in close communication about design and implementation. You review each other’s code and you rely on “individuals and interactions over process.”

With the new investment, you’re getting lots done -- and your single codebase might be starting to get unwieldy. Perhaps the app module’s build.gradle is getting longer and more difficult to keep organized. You’re probably also thinking about which of your components can be re-used and which are the best ways to do it. Maybe some of those utilities are even generic enough that they could be useful beyond your immediate project. You’ve got some decisions to make.

Firstly, should you split the utility code out into a new library module? Generally speaking, this is a good idea if you’ve got a well-designed public contract that’s being exercised by 3 or more other components. It’s also a pretty minor change to make, and with little downside. The primary gain here will be to simplify your build script and to create a more explicit barrier between groups of interacting components.

Creating re-usable library modules will also help reduce your build times, since Gradle will be able to recycle any already-built modules that haven’t been touched. There are a number of steps needed to produce an apk, but re-usable library modules can (at minimum) chip away at the incremental compilation work that’s needed to construct the final apk.

With a bonafide library module in your project, you’ve unlocked an additional capability. You can now release your library module as a public Android ARchive. Android libraries such as OkHttp or Glide are released this way, and are distributed via the Maven Central Repository (or Google’s mirror of it.) And just as is the case with OkHttp or Glide, your app can depend on a remote copy of your prebuilt library instead of directly on the local module code.

There are a few benefits to doing this. Even with a local library project, you have to do at least one initial build to create the local library artifact. Not so with an external library. Now, you just pull a built artifact and skip that entire compilation unit.

By the time you’re publishing a library artifact and your app depends on it as an external artifact, there’s little benefit to keeping it in the same Gradle project. So, you can further simplify things by splitting the repo into two Gradle projects entirely.

This allows you to operate on a smaller application project, skipping some compilation, and with some code out-of-sight and out-of-mind.

Act 4: An Enterprise Monorepo

Few organizations ever grow beyond “Act 3.” But there are several additional challenges that creep up, for those that do. Your team might have some of the problems below -- or perhaps you already have all of them.

Firstly, an "enterprise monorepo" is no longer the collective effort of just a few developers. Enterprise monorepos may contain hundreds of thousands of lines of code -- sometimes millions of lines of code. You might have dozens of Engineers working in different parts of the company, all trouncing on the same repo. Each engineer brings different priorities, perspectives, and proficiencies. You’ll have multiple managers leading different teams, reporting out delivery schedules.

With more code and more developers, the obstacles at this phase are both technical and social. One central questions becomes “how do we break apart the codebase so that different teams can function independently?”

You need to start creating tooling to assign ownership. One solution is to use a CODEOWNERS file, to enforce required approvals when different parts of the repository are touched.

Even more-so than when the app was small, your build times are probably getting bad. It becomes technically-complex to reduce the footprint of the top-level application module, and feature teams may not be incentivized to invest into that effort.

At this point, there are two key technical strategies you can employ to mitigate this.

Firstly, an authoritative voice must decree that all feature teams shall own and publish prebuilt, strongly-versioned library modules. In some-senses, this is moving your organization towards a Mobile-Service-Oriented-Architecture of sorts. (The key difference being that services talk over HTTP, whereas your libraries talk across compilation units.) This strategy yields three key benefits:

The feature library may be consumed and developed by a team-specific top-level application -- a lightweight “dev app" or “sample app." This lightweight app will have limited functionality, and be much easier to work with than the main app itself.
The feature library may be consumed by a different sort of top-level actor: an integration test harness. This will provide a new integration-test entry point.
When the main application consumes the feature library, it will not incur the cost of recompilation.

As with SOA, the details of “how” a library contract is fulfilled is left to the responsible team to determine.

The second approach you can utilize to mitigate app-creep is to invest in an extensibility programme. Namely, you need to develop an application architecture that enables feature teams to build outside of the existing application project.

Act 5: Executing an Extensibility Programme

In order for a feature to live entirely outside of the app module, it needs to be able to call the various dependencies it needs to function -- none of which can be in the app module. In fact, it is impossible for the library module to call back into the app directly: this would create a cyclical dependency.

The first phase of your extensibility program should be to identify the “big lists of things” files. Every app has them. They often materialize as big lists of constants - a routing table, perhaps. The problem with these types of files is that a feature module will necessarily need to integrate with them -- and so will necessarily need to have some degree of footprint in the app.

Such files are usually designed without the open-closed principle in mind. (The author wasn't naive, they were just designing a smaller-sized app at that point. And in that context and time, they actually did the right thing.)

You can often rework such components by either:

Code-generating them. To do this, create build tool that sources input fragments from multiple source repositories.
Adding run-time pluggability. To do so, make an interface for feature components so that they can addPlugin() and removePlugin() (or addRoute(), removeRoute(), etc.)

The other key strategy is to use Inversion of Control (IoC). IoC became popular in large service monoliths. And guess what? We necessarily are building a monolith! At the end of all of this, there's only one app going to the Play Store. IoC dependency containers enable components to speak to one another over abstractions (interfaces), without knowing the details of how things actually will be carried out.

In the diagram above, the fictitious “Pizza domain” library needs to use the View/UI contacts so that it can initiate the display of some UI screen. To do so, the Pizza domain takes a dependency on the View/UI contract library. When the application module pulls in its various dependencies, it assigns a concrete implementation of the view/UI contract via service binding.

Each vertical feature (in yellow) publishes a contract library and an implementation library. The application module itself uses a Service Locator to create a binding between contract interfaces and concrete implementations there-of. Ordinarily, this binding code is owned by a supervisory team that actually releases the app, so they can police the list and enforce social policies over it during code review. (In the example above, the binding code lives in the app itself, but could be further isolated into an independent library.)

Act 6. Further Divisions of Source Code

At this phase, we have numerous independently-versioned libraries being published. We have an extensible application architecture that has helped us to prevent growth in our application module itself. Builds are generally faster, and teams can develop and test their features without needing to use the main application module all the time.

By now, you still have lots of thrash on your monorepo. You’re still using CODEOWNERS to help with change control. Should you continue dividing the project further, into separate Git repos? That would surely lead to one of the most decoupled and distributed codebases achievable, right?

There’s overhead in setting up all of these independent repositories, yea, but you could potentially script their creation with some Infrastructure-as-Code-type-tooling. After a while, many of these repositories will be low-touch. Out-of-sight, out-of-mind. Stable APIs will form, and new changes will cease to come in.

The more repositories, Gradle projects, and Gradle modules you create, the more overhead there will be to get work done. But, your teams will be fully out of each other’s way. For the most part.

Handling Dependency Updates

As your application proceeds through these phases of life, you’ll find that managing dependency upgrades will be a pain. When everything is all in one place, you could rely on Herculean efforts to upgrade the single code base. One all-nighter, one Engineer, block out some time on everyone’s calendar, and get the Pull Request in.

That's less palpable as your organization grows bigger. You need additional social mechanisms to plan for changes, and you won’t be able to make them as quickly.

One workflow to execute dependency upgrades across teams might be to broadcast "need by" dates, and ask teams to provide the version numbers of their libraries that will institute the upgrade. Then, the supervisory team (app release team) can create a single PR which updates the dependency versions.

Closing Thoughts

As your app and team gets bigger, there is generally more overhead required to add more structure to it. Don't use a Jackhammer on drywall, and don't use a putty knife on concrete. Only decompose your application when you hit the next incremental limit of scale.

Happy architecting.

Unboxing the Qualcomm RB5 Robotics Kit

Jameson — Mon, 19 Apr 2021 00:49:49 +0000

Today I'll be showing some of my very first interactions with a Qualcomm RB5 Robotics Kit. In their words:

The Qualcomm Robotics RB5 development kit combines the promise of 5G with the computing power for AI, deep learning, computer vision, 7-camera concurrency, rich multimedia and security

Inside the box, we get three parts:

The board itself,
A USB-C cable,
A power brick.

The bringup guide tells us we'll need a Linux workstation, and a few utilities from the Android SDK: adb & fastboot. In short, it looks like a familiar programming model:

Build device images on your local machine;
Deploy and run those images on the device.

I'm on a Mac right now, so I won't have access to the full toolset needed to prepare the images. However, I've got a working adb and fastboot, so can easily interact with the device. This will likely continue to be my workflow, even after I get a Linux build host going in EC2.

Let's get the kit running. I connect the power brick, and then I connect the USB-C cable into my MacBook. After a bit, a green LED indicator lights up on the device:

At this point, I'm able to see the device via adb:

$ adb devices
List of devices attached
ZTR10S1600CS    device

I can even login and probe for some details:

$ adb shell
sh-4.4#

The device is running Ubuntu 18.04.05:

sh-4.4# lsb_release -a 
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:    18.04
Codename:   bionic

The processor has 8 cores:

sh-4.4# nproc 
8

It's an AArch64:

sh-4.4# cat /proc/cpuinfo | grep ^Processor 
Processor   : AArch64 Processor rev 14 (aarch64)

sh-4.4# uname -a 
Linux qrb5165-rb5 4.19.125 #1 SMP PREEMPT Sat Mar 20 11:48:10 CST 2021 aarch64 aarch64 aarch64 GNU/Linux

The board has 7650MB of memory:

sh-4.4# free -m
              total        used        free      shared  buff/cache   available
Mem:           7650        1112        6058          12         478        6848
Swap:          3825           0        3825

There are a few USB devices on the board, an AX88179 ethernet controller, and a couple of USB hubs:

sh-4.4# lsusb 
Bus 002 Device 003: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
Bus 002 Device 002: ID 05e3:0625 Genesys Logic, Inc. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Not much on the PCI bus:

sh-4.4# lspci 
00:00.0 PCI bridge: Qualcomm Device 010b (rev ff)
01:00.0 Unassigned class [ff00]: Qualcomm Device 1101 (rev ff)

Lots and lots of partitions:

sh-4.4# mount 
/dev/sda7 on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2671320k,nr_inodes=667830,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/schedtune type cgroup (rw,nosuid,nodev,noexec,relatime,schedtune)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/debug type cgroup (rw,nosuid,nodev,noexec,relatime,debug)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sde9 on /dsp type ext4 (ro,nosuid,nodev,noexec,noatime,discard,noauto_da_alloc,data=ordered)
/dev/sde4 on /firmware type vfat (ro,nodev,noexec,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sde5 on /bt_firmware type vfat (ro,nodev,noexec,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)

Looks like a 128GB storage chip, split apart into different partitions:

sh-4.4# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        99G  6.0G   93G   7% /
devtmpfs        2.6G     0  2.6G   0% /dev
tmpfs           3.8G     0  3.8G   0% /dev/shm
tmpfs           3.8G   12M  3.8G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
tmpfs           3.8G     0  3.8G   0% /var/volatile
/dev/sde9        59M   24M   34M  42% /dsp
/dev/sde4       395M   61M  335M  16% /firmware
/dev/sde5        64M  368K   64M   1% /bt_firmware

We've found some of the components in the hardware spec sheet so far, but just a few. Of particular interest, I'd like to locate some of the coprocessors (the "heterogeneous computing" bit):

Qualcomm Adreno 650 GPU
Qualcomm Hexagon DSP
Qualcomm Spectra 480 image processing
Adreno 665 VPU video encode/decode
Qualcomm Secure Processing Unit SPU240
Qualcomm Neural Processing unit NPU230

In an upcoming article, we'll start trying to exercise some of these components!

Android WebSocket Clients for Amazon API Gateway

Jameson — Fri, 08 Jan 2021 09:25:05 +0000

WebSockets are a great way to achieve bi-directional communication between a mobile app and a backend. In December 2018, Amazon's API Gateway service launched serverless support for WebSockets.

Setting it all up is somewhat involved. I recommend starting with the Simple WebSockets Chat backend they provide as a demo. We'll focus on how to talk to it from Android, using Square's familiar OkHttp or JetBrains' newer Ktor.

Pre-requisites

An activate an AWS account
The latest version of the AWS CLI
wscat installed via npm.
AWS SAM CLI installed via Homebrew (if on Mac.)
Android Studio

Setting Up a Backend

Installing the serverless demo app is pretty straight-forward.

Checkout the Chat app:

git clone https://github.com/aws-samples/simple-websockets-chat-app.git

And deploy it to your account:

sam deploy --guided

After a while, that command will finish. You'll be able to see the provisioned resources by inspecting the outputs of the CloudFormation stack:

aws cloudformation describe-stacks \
    --stack-name simple-websocket-chat-app \
    --query 'Stacks[].Outputs'

It creates a DynamoDB table, three Lambda functions, an AWS IAM role, and an API Gateway with WebSockets support.

After you've glanced over the output to understand what all was created, let's zero in on the WebSocket endpoint itself. We'll need to find the URI that our client can use, to access it.

aws cloudformation describe-stacks \
    --stack-name simple-websocket-chat-app \
    --query 'Stacks[].Outputs[]' | \
jq -r '.[]|select(.OutputKey=="WebSocketURI").OutputValue'

This should output something like:

wss://azgu36n0vf.execute-api.us-east-1.amazonaws.com/Prod

Command Line Testing using `wscat`

Let's quickly test the backend, to make sure it works as we expect. We'll use the wscat command-line utility. The command below will open a long-lived connection to the API Gateway, and we'll be able to send and receive messages in a subshell:

$ wscat -c wss://azgu36n0vf.execute-api.us-east-1.amazonaws.com/Prod
Connected (press CTRL+C to quit)
> {"action": "sendmessage", "data": "foo"}
< foo

The JSON format above is required by the app we deployed. You can change the value of "foo", but you can't change anything else.

If you try to pass something else, it doesn't work:

> Hello, how are you?
< {"message": "Forbidden", "connectionId":"Y0bEuc0UIAMCIiA=", "requestId":"Y0bwuGjXIAMFmEg="}

But, if you open up multiple terminal windows, and connect them all to the endpoint, they'll all receive a valid message.

Calling the API from Android via OkHttp

Now that we know the WebSocket API is working, let's start building an Android app to use as a client, instead.

WebSockets have been supported in OkHttp since 3.5, which came out all the way back in 2016.

Initializing a WebSocket client is straight-forward:

val request = Request.Builder()
    .url("wss://azgu36n0vf.execute-api.us-east-1.amazonaws.com/Prod")
    .build()
val listener = object: WebSocketListener() {
    override fun onMessage(ws: WebSocket, mess: String) {
        // Called asynchronously when messages arrive
    }
}
val ws = OkHttpClient()
    .newWebSocket(request, listener)

To send a message to the API Gateway, all we have to do is:

ws.send(JSONObject()
    .put("action", "sendmessage")
    .put("data", "Hello from Android!")
    .toString())

We can add a button to our UI with ViewBinding, and fire the WebSocket message whenever we click it:

ui.button.setOnClickListener { 
    ws.send(JSONObject()
        .put("action", "Hello from Android!")
        .put("data", command)
        .toString())
}

Since all of the threading is handled inside OkHttp, there really isn't a lot more to it. Save a handle to your view binding and to to your WebSocket client when you create your Activity:

class MainActivity : AppCompatActivity() {
    private lateinit var ui: ActivityMainBinding
    private lateinit var ws: WebSocket

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        ui = ActivityMainBinding.inflate(layoutInflater)
        setContentView(ui.root)
        connect() // As above
    }

Calling the API from Android via Ktor

Ktor is built with the assumption that you're using Coroutines, and managing your own scope/context. This makes it a more flexible tool, but adds some additional complexity.

The basic setup for the tool is going to look like this:

private suspend fun connect(ktor: HttpClient, u: Url) {
    ktor.wss(Get, u.host, u.port, u.encodedPath) {
        // Access to a WebSocket session
    }
}

private /* not suspend */ fun connect() {
    val url = Url("wss://azgu36n0vf.execute-api.us-east-1.amazonaws.com/Prod")
    val ktor = HttpClient(OkHttp) {
        install(WebSockets)
    }
    lifecycleScope.launch(Dispatchers.IO) {
        connect(ktor, url)
    }
}

Inside the trailing closure, you have access to an instance of DefaultClientWebSocketSession. It has two important members:

A ReceiveChannel named ingoing, and
A SendChannel named outgoing.

SendChannel and ReceiveChannel are the inlet and outlet to a Kotlin Channel, which is basically like a suspendible queue.

It's pretty trivial to send and receive some simple messages inside the WebSocket session closure:

ktor.wss(Get, u.host, u.port, u.encodedPath) {
    // Send a message outbound
    val json = JSONObject()
        .put("action", "sendmessage")
        .put("data", "Hello from Android!")
        .toString()
    outgoing.send(Frame.Text(json))

    // Receive an inbound message
    val frame = incoming.receive()
    if (frame is Frame.Text) {
        ui.status.append(frame.readText())
    }
}

However, we're missing a bunch of functionality that we had in our OkHttp solution. Namely:

We want to send and receive data at the same time, in a loop, and
We want to get notifications when the connection opens and closes.

Parallel send and receive in Ktor

Our goal here is ultimately to dispatch a stream of messages to the outgoing channel, and to consume a stream of messages from the ingoing channel, at the same time. The basic approach we'll use is to launch two bits of work asynchronously, and then wait for them to finalize:

ktor.wss(Get, u.host, u.port, u.encodedPath) {
    awaitAll(async {
        // Code that will send messages
    }, async {
        // Code that will receive messages
    })
}

We need some stream of events to trigger the send messages. Adding a button element to our UI makes sense. But, we'd like to model the button clicks as a stream of commands, by the time we dispatch events over the WebSocket.

Let's first create an extension function to map button clicks into a Flow<Unit> (credit to StackOverflow):

private fun View.clicks(): Flow<Unit> = callbackFlow {
    setOnClickListener { offer(Unit) }
    awaitClose { setOnClickListener(null) }
}

Now, we can listen to a flow of events from a button, map them into the format we need, and send them:

ui.button.clicks()
    .map { click -> "Hello from Android!" }
    .map { message -> JSONObject()
        .put("action", "sendmessage")
        .put("data", message)
        .toString()
    }
    .map { json -> Frame.Text(json) }
    .collect { outgoing.send(it) }

That will work well for the outgoing events. Now, we just need to respond to inbound events, in the second async block.

In this case, there isn't a lot of value to mapping the result. The value we receive over the socket is the contents associated with the "data" key in the messages. So for example, we might get "Hello from Android!":

incoming.consumeEach { frame ->
    if (frame is Frame.Text) {
        ui.status.append(frame.readText())
    }
}

When it's all said and done, we end up with something like this:

ktor.wss(Get, u.host, u.port, u.encodedPath) {
    awaitAll(async {
        ui.button.clicks()
            .map { click -> JSONObject()
                .put("action", "sendmessage")
                .put("data", "Hello from Android!")
                .toString()
            }
            .map { json -> Frame.Text(json) }
            .collect { outgoing.send(it) }
    }, async {
        incoming.consumeEach { frame ->
            if (frame is Frame.Text) {
                ui.status.append(frame.readText())
            }
        }
    }
})

Ktor's lifecycle events

OkHttp allowed us to override callbacks on the WebSocketListener, to get notified of various lifecycle events:

val listener = WebSocketListener() {
    override fun onOpen(ws: WebSocket, res: Response) {
        // when WebSocket is first opened
    }
    override fun onClosed(ws: WebSocket, code: Int, reason: String) {
        // when WebSocket is closed
    }
}

Ktor doesn't work like that. They suggest some different approaches to recover those events.

The easiest one to recover is the event when the socket opens. It's just the first thing that happens inside of the wss session closure:

ktor.wss(Get, u.host, u.port, u.encodedPath) {
    ui.status.append("Connected to $u!")
}

To get more insight into the WebSocket termination, we can expand our processing in our receive block:

incoming.consumeEach { frame ->
    when (frame) {
        is Frame.Text -> { /* as above */ }
        is Frame.Close -> {
            val reason = closeReason.await()!!.message
            ui.status.append("Closing: $reason")
        }
    }
}

Catching failures is also fairly easy:

private fun connect() {
    val url = Url("wss://azgu36n0vf.execute-api.us-east-1.amazonaws.com/Prod")
    val client = HttpClient(OkHttp) {
        install(WebSockets)
    }
    lifecycleScope.launch(Dispatchers.IO) {
        try {
            connect(client, url)
        } catch (e: Throwable) {
            val message = "WebSocket failed: ${e.message}"
            ui.status.append(message)
        }
    }
}

Wrapping Up

Well, there you have it: a rough and dirty explanation of how to use OkHttp and Ktor to consume an Amazon API Gateway WebSockets API. 🥳.

The code for this project is available on my GitHub page.

A Brief History of AWS Mobile SDKs and How to Use Them in 2021

Jameson — Sat, 26 Dec 2020 23:11:27 +0000

In this article, I'm going to contextualize some history of AWS' mobile products, and explain when and why you should (or should not) use them.

Early Development of Android support at AWS

This year, the AWS SDK for Android turned 10 🥳. The Android SDK originated as a fork of the original Java SDK.

To frame the discussion, let's recall what AWS looked like in 2010. S3 and EC2 were among our only offerings. Each was still in its infancy of public availability.

2010 was the start of the "Web 2.0" revolution, wherein it was no longer enough for a company to be "online." In the Web 2.0 world, you needed to have an API, too.

In response to the dramatic uptick in demand for APIs, REST began to overtake SOAP as the dominant architectural pattern for client-server exchange on the public Internet. Then-recently created OAuth emerged as the dominant standard for authenticating service requests.

A key point here is that the Android SDK was envisioned before these practices had emerged, not in response to them.

Overview of AWS SDKs

Shifting gears for a moment, let's talk about the AWS SDKs in general. AWS offers SDKs for JavaScript, Python, PHP, .NET, Ruby, Java, Go, Node.js, C++. I'm aware of active projects to support at least three more popular modern languages.

We also provide platform-focused SDKs, like the SDKs for Android, iOS, and front-end JavaScript.

One immediate question you might have as a front-end dev is:

Why wouldn't you just use the Java SDK for Android, and the JavaScript SDK for the web? And the (er) ... what about iOS?

Why You should use our Mobile SDKs

There are a number of reasons why you might want to use one of our mobile SDKs (as opposed to a traditional SDK.)

First off, if you're an iOS developer, you don't really have an option right now. The iOS SDK is currently the only SDK that supports Objective C or Swift.

But even on Android, the V2 Java SDK uses some advanced language features, that are not supported on Android. For backend developers, it makes sense to take full advantage of modern JVM features in a modern Java SDK. Unfortunately, the tradeoff is that the Java SDK isn't compatible with Android's runtime, "ART."

Another reason to prefer the mobile SDKs is that they tend to be vastly stripped down in comparison to the traditional SDKs. For reference, the Android SDK has tens of modules, many of which are focused on data-plane operations. The V2 Java SDK has hundreds of fully-featured modules, useful in managing server infrastructure. It would be wasteful to include all of that extra bloat on a resource constrained device.

Lastly, AWS Amplify also provides some high-level client libraries which are specifically tailored to use cases commonly encountered on the front-end. For example, most apps need to handle user authentication, save data to the cloud, and record some usage metrics.

The high-level Amplify libraries are plug-and-play for developers without a lot of experience fiddling with the AWS backend. If you're just starting out with an idea, and want to build it out quick: use Amplify. You'll be able to think about the problem in an idiomatic way, using workflows familiar in your front-end development environment.

Why you should not use the Mobile SDKs

"Great!" you say. So why wouldn't I use the mobile SDKs?

The answer to this draws back to my opening point about AWS' history. The SDKs were designed at a time when it was still future-looking to simply communicate with a remote endpoint over REST. That ship has sailed. That's how 95% of new APIs work, now.

Let's distinguish between two types of client-server interactions. There is a "north-south" interaction, where a public client talks to your service over the Internet, and there is an "east-west" interaction, where your services talk to each other behind a gateway.

I posit that the AWS SDKs are primarily used for server-to-server communications, to toggle control/admin plane functionalities. In its infancy, AWS prescribed no mechanisms for the data plane. That was left to whatever application you installed in EC2.

Why does it matter? Mobile devices are always a "north-south" interaction. Once you release your application to the App or Play Store, you lose control over them.

In a perfect world, we could just implement business logic, and make it available for use over the public Internet. But, we don't live in a perfect world, alas. APIs that serve the public Internet need to be conscious of DDoS attacks, they need to bill/meter resource consumption, throttle over-zealous actors, authenticate and authorize requests from different parties. The list of additional functionalities starts to go on and on.

In fact, there are so many additional features needed, even before we reach your business logic, that it really makes sense to put all of this stuff in its own "front-end service." At Amazon, we have only a handful of services that implement this full range of "front-end" functionalities. I'll call these "mobile-blessed services." The leader of the pack is Amazon API Gateway.

How You Really Ought to Use SDKs with AWS

Your mobile applications should interact with your business logic through a RESTful endpoint managed by Amazon API Gateway. You should prefer not to make direct calls to most Amazon services directly from a mobile device.

This mantra has some big implications for the utility of the mobile SDKs. As one arbitrary example, let's consider a use case where we want to call Amazon Translate, to translate some text. Instead of using the Amazon Translate support in the iOS or Android SDKs, I'm suggesting that you should pick any supported language, and implement those Translate service calls behind the API Gateway. Render an API response that contains only the bits you need on the mobile device.

By the time you've implemented this solution, you don't need much on the client. You need a tool to auth users, and a way to sign requests to Amazon API Gateway. Retrofit and Volley are de facto standard REST client libraries.

Now, there are a lot of different things you can put behind Amazon API Gateway. But for most small-to-medium sized applications, where you don't want to futz about with EC2 instances or containers, you should probably start with an AWS Lambda.

Exceptions to the Rule & Special Snowflakes

AWS has a few other mobile-oriented services and/or services with a rich set of "front-end" features: Cognito, S3, CloudFront, AppSync. Similarly, any discussion of various AWS services would be replete without guidance around EC2 and DynamoDB.

Cognito is almost a prerequisite to any serious AWS mobile integration. It provides mechanisms to hook into Google, Apple, and Facebook accounts. You can assign rules around which users get to access what AWS resources. There is a lot of client side logic needed to implement a secure session with Cognito. I recommend using our high-level Amplify Auth for this.

S3 is actually Amazon's oldest service. It hooks into CloudFront. It has been designed explicitly to serve low-latency content over the public internet. S3, combined with a CDN of some kind, is absolutely a "front-end" service.

AppSync is a newer offering, providing a serverless, managed GraphQL endpoint. My team is actively investing a huge amount of energy into the product space. Overall, this is still emerging technology at Amazon. AppSync lacks some basic P0 front-end functionalities like throttling.

EC2 is one of our most popular services. However, there are limited use cases to provision servers directly from a mobile handset over the public Internet. This a prime example of something you should not be doing with the AWS Mobile SDKs.

Dynamo is in a similar boat. It's a well-known bad practice to call a database directly from an application. However, Dynamo is arguably one of the most useful services AWS provides. We've built an entire front-end product based on transacting data models into and out of Dynamo. Please checkout the Amplify DataStore!

Wrapping Up

Above, I've tried to provide a bit of context on why we have the products we do, and how you might like to use them. I've laid out some strong opinions. But, as with all solutions in Engineering, you need to look at your use case, and weigh the tradeoffs. I hope this article has moved you closer to a good solution in your unique situation. Happy hacking!

Nightly GitHub Contribution Stats with PyGitHub and AWS Lambda

Jameson — Sat, 26 Dec 2020 01:27:57 +0000

// Detect dark theme var iframe = document.getElementById('tweet-1342446011357290497-855'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1342446011357290497&theme=dark" }

Today I'm going to show you how to build a GitHub stats scraper using a scheduled-execution AWS Lambda function.

The Python code inside the Lambda will leverage PyGitHub to communicate with the V3 GitHub APIs. The code builds a simple Markdown-flavor report, then publishes it to a Gist. The Gist will display the Markdown in rendered form, so you can share a link to it.

The tables we'll build will look something like this:

Motivation

AWS Amplify is almost entirely open-source. We use GitHub to host our code. Our engineers tend to work across our various repositories, and it can be tough to keep track of contributions. Likewise, we want to recognize external parties who have donated their time and mindshare.

Some questions I'd like this tool to help answer:

Who are the top authors of pull requests, across our GitHub org?
Who are the top reviewers of pull requests, across our GitHub org?

There are, of course, lots of ways to contribute to a project. Tracking activity on GitHub pull requests is just one view of the world. This tool just makes data available. It doesn't prescribe a way for you to analyze it.

System Overview

The system works like this:

Amazon CloudWatch fires a periodic event
It causes an AWS Lambda function to execute
The lambda function queries GitHub's API for contribution data
The lambda builds a Markdown report
The lambda publishes the Markdown to a Github gist

Setting up the Basics

Creating a lambda function and setting it up for periodic execution are fairly straightforward tasks in the AWS Console. After you follow the linked guides from the AWS documentation, you should be left with a dummy Python function that just prints out whatever arguments are passed to it.

In order to start communicating with GitHub, we'll have to obtain an access token. The PyGitHub documentation shows a concise example of initializing a client with a GitHub token:

g = Github('access_token')

To obtain one, head over to github.com/settings/tokens, and click "Generate new token" in the top-right. Give it some name you'll remember. Leave all of the scopes unchecked, but do check the gist box:

Lastly, click "Generate token" at the bottom of the page, and copy-paste the output. Keep the token someplace safe where only you'll have access to it.

Setting up Dependencies in our Lambda

For simple computations, you can often update a Lambda function directly from the AWS Lambda Console. However, PyGitHub doesn't exist in the Lambda execution environment. So we'll need to make it available to our Lambda, somehow. The strategy we'll use is to bundle all of our dependencies with the Lambda, so that they're locally available when the code runs.

AWS Lambda provides documentation to "Deploy Python Lambda functions with .zip file archives." But, I'll take a more targeted approach, here. The tasks needed to add the PyGitHub dependency to our Lambda are described below.

First, use the AWS CLI to find the lambda function you created:

aws lambda list-functions | \
    jq -r '.Functions[].FunctionName'

This outputs:

ExampleLambda
demo_function
top_contribs_periodic

In this list, I recognize top_contribs_periodic as the Lambda that I created earlier.

Now we're going to ask AWS Lambda to give us a URL from which we can download the function code:

aws lambda get-function \
    --function-name top_contribs_periodic | \
    jq -r '.Code.Location'

The URL will be very long. It's a pre-signed S3 URL for a zip file in an AWS-managed S3 bucket. It should start off like:

https://prod-04-2014-tasks.s3.us-east-1.amazonaws.com/snapsh....

Now we can copy-paste that URL, and use curl to download the zip file from S3:

curl '<paste-url-from-above>' --output original_lambda.zip

While we're working on this, we're going to be manipulating zip files that contain a bunch of content at the root level of the archive. Whenever I'm working with a zip that doesn't have a single top-level directory in it, I like to know what I'm dealing with, first:

zip --show-files original_lambda.zip

This outputs:

Archive contains:
  lambda_function.py
Total 1 entries (2217 bytes)

Let's make a working directory, and unzip the code there:

mkdir workingdir
cd workingdir
unzip ../original_lambda.zip

Next, we're going to use the virtualenv utility to install our dependencies into this directory. (This step follows the Lambda documentation, "with a virtual environment" pretty closely.)

Let's make sure you have virtualenv installed:

pip3 install virtualenv

Now, create a virtual environment:

virtualenv myvenv

Activate the environment:

source myvenv/bin/activate

Install PyGitHub:

pip install pygithub

Almost done. Deactivate the environment:

deactivate

Start creating a new zip file. We'll upload this zip back to AWS Lambda, in a second.

cd myvenv/lib/python3.8/site-packages
zip -r ../../../../updated_lambda.zip .

The last thing we need to do before re-uploading the code is to package our lambda_handler.py back into the .zip. So that we can validate our dependency installation, let's include a very simple use of the PyGitHub. Update the lambda_handler.py so that it looks like this:

import os
from github import Github

def lambda_handler(event, context):
    g = Github(os.environ['GITHUB_TOKEN'])
    g.get_user()

The code above will look for your GitHub token in the environment, try to instantiate an API client, and the call some simple API, and not crash.

Save this file into the zip:

cd -
zip -g updated_lambda.zip lambda_handler.py

And finally, upload it:

aws lambda update-function-code \
    --function-name top_contribs_periodic \
    --zip-file fileb://updated_lambda.zip

Back in the AWS Console, add a new environment variable, GITHUB_TOKEN, into your Lambda function. Set its value to the one you captured from github.com/settings/tokens.

By now, you should be able to click the "Test" button in the top-right of your AWS Lambda console, to try and invoke the function. The simple Lambda should succeed. But of course, it doesn't really do much yet.

Building out the Core Logic

Well, we succeeded in making a simple call to GitHub from AWS Lambda! Great. Now, let's implement some logic.

Lets update the Lambda to do something a little more complex.

import os
from github import Github
from datetime import *

def list_recent_contributions(g):
    org = g.get_organization('aws-amplify')
    repos = [r for r in org.get_repos()]
    month_ago = datetime.now() - timedelta(30)
    for pull in repo.get_pulls(sort='created', direction='desc', state='all'):
        if pull.created_at < month_ago:
            break
        login = pull.user.login
        title = pull.title
        print('Recent pull from {0}: {1}'.format(login, title))

def lambda_handler(event, context):
    g = Github(os.environ['GITHUB_TOKEN'])
    list_recent_contributions(g)

The code above finds all public repos in an organization ('aws-amplify'), and then queries them for pull requests. The search is in descending order by creation date, meaning that recently created PRs show up first. We stop iterating over the pages of results when we start seeing pulls that are outside the window in which we're interested. (In this case, we're only looking at the last 30 days.)

You can kind of see the possibilities at this point. You can bounce between the PyGitHub Library Reference and the GitHub REST API reference and start tuning the logic to meet your needs.

Building Markdown

There are two main challenges in producing the Markdown report.

Decoupling document text from Python language syntax
Producing data tables, row by row

To achieve the first goal, I'm using a mutli-line string. To strip the leading indentation in the Python document, I call textwrap.dedent on it. The document text has a couple of placeholders that I populate with str.format(...), at the end. My documentation creation function looks roughly like this:

    return textwrap.dedent("""
    # Top Contributors, Last 30 Days

    This is a list of the top contributors to the aws-amplify Github org's public repos.
    Contributions from the last 30 days are considered.
    This document is updated by a cron job every day.
    Contributors are from AWS and from the community.
    Contribution counts are a running sum of a user's contributions across all repos.

    ### Top 10 Authors
    {0}
    ### Top 10 Reviewers (by total comments)
    {1}

    -----------------------

    Last updated {2}.

    """).format(authors_table, reviewers_table, str(datetime.datetime.today()))

But what about the authors_table and reviewers_table? How are those obtained? To build the Markdown tables, I use some more string templates. I have a utility function which can transform a dict into a Markdown table:

def top_ten_table(key_label, val_label, entries):
    row_template = '|{0}|{1}|{2}|\n'
    table = row_template.format('Rank', key_label, val_label)
    table += row_template.format('--------', '--------', '--------')
    item_list = sorted(entries.items(), key=lambda x: x[1], reverse=True)
    for index, (key, value) in enumerate(item_list[:10]):
        table += row_template.format(str(1 + index), key, str(value))

    return table

Publishing a Gist

The last big piece of this project is to publish the outputs somewhere.

There are a number of cool possibilities here. If you're using some static hosting solution like Amplify Hosting or GitHub Pages, your published Markdown could get stylized with whatever rules your existing web app applies. You could pretty much publish the Markdown anywhere.

I had considered publishing the output to my GitHub Pages repository, and then letting my Jekyll site render it with its stylesheet.

But, in the interest of keeping things simple, let's just put it into a Gist for right now.

The Gist APIs are actually a weak-spot in the PyGitHub implementation, IMO. What I'd like to do is just have a user.update_gist(...) function available. Unfortunately, there isn't anything exactly like that. Instead, we have to related capabilities available:

Search all of my Gists, get a handle to one, and call .update(...) on it;
Create another new Gist.

Well, fine. Let's try to encapsulate those two things into a single utility method:

def write_gist(gh, filename, description, content):
    files = {filename: github.InputFileContent(content=content)}
    user = gh.get_user()
    print("Looking for matching Gists....")
    for gist in user.get_gists():
        if gist.description == description:
            print("Found a matching Gist. We'll updated it.")
            gist.edit(files=files, description=description)
            return
    print("No existing Gist, creating a new one....")
    user.create_gist(public=True, files=files, description=description)

This function will look for an existing Gist with a given description. If it finds one, then it updates it. If it doesn't find one, it continued to create a new Gist with that description. 🥳

When you run this function, it will save the provided content to Gist. Since it currently hard-codes 'contrib.md' as the filename, the resulting URL you end up with will look like this:

https://gist.github.com/yourusername/uniqueid#file-contrib-md

Wrapping Up

That's most of the nuts and bolts to it. The complete script I'm using is available here, on GitHub.

The output of that script is visible here.

Let me know what you think! What features should I add to it next? Happy hacking, and happy holidays. 🎄

DEV Community: Jameson

Pre-push Hooks

What is a Hook?

Why Not Just CI?

Pre-commit vs. Pre-push

How to build a pre-push system

The installation script

The pre-push script

A pre-push.d script

Conclusion

Should You Adopt a Cross-Platform Client Framework?

What Problem Is Being Solved?

How Tech Businesses Evolve

What Are the Options?

React Native

Flutter

Kotlin Multiplatform

Shared Architecture, Separate Platforms

When to Choose Each

Wait, What About Tech Factors?

Modern Async Primitives on iOS, Android, and the Web

Let's Talk About Asynchronicity

Swift: async/await & AsyncSequence

Kotlin: Coroutines, Flow

JavaScript: aysnc/await, Promises, AsyncIterator/AsyncGenerator

Wrapping Up

Splitting SwiftData and SwiftUI via MVVM

First Attempt at MVVM & SwiftData

Updating App integration

Defining a Data Source

Simple Runtime Feature Gating on Android

Fakes & Mocks on Android: Well Partner, that depends.

An Appeal to the Classics

How Sociological Trends Work

"Mocks Require Mock-able Code"

Reflection is Slow and Spooky

Fakes Have an Upfront Cost

"Fakes Have Lower Maintenance Cost"

Wrapping up

Bash Completion for Git on Mac OS X Monterey

Quick background on bashrc files

jamesonwilliams / dotfiles

Jameson's . (dot) files for Linux

dotfiles

Setup

How it used to work on Mac OS X

How it works now

Scaling Development of an Android App

Act 1: The Simple Sample App

Act 2: The Prototype That “Made It”

Act 3: A Real Production App

Act 4: An Enterprise Monorepo

Act 5: Executing an Extensibility Programme

Act 6. Further Divisions of Source Code

Handling Dependency Updates

Closing Thoughts

Unboxing the Qualcomm RB5 Robotics Kit

Android WebSocket Clients for Amazon API Gateway

Pre-requisites

Setting Up a Backend

Command Line Testing using wscat

Calling the API from Android via OkHttp

Calling the API from Android via Ktor

Parallel send and receive in Ktor

Ktor's lifecycle events

Wrapping Up

A Brief History of AWS Mobile SDKs and How to Use Them in 2021

Early Development of Android support at AWS

Overview of AWS SDKs

Why You should use our Mobile SDKs

Why you should not use the Mobile SDKs

How You Really Ought to Use SDKs with AWS

Exceptions to the Rule & Special Snowflakes

Wrapping Up

Nightly GitHub Contribution Stats with PyGitHub and AWS Lambda

Motivation

System Overview

Setting up the Basics

Setting up Dependencies in our Lambda

Building out the Core Logic

Command Line Testing using `wscat`