DEV Community: Charlie Foxtrot

How to build a MAUI app in Azure Devops

stfnilsson — Tue, 29 Oct 2024 07:50:25 +0000

- and how to publish it to Test Flight and Google Play Console

You have created a .NET MAUI app but you have been building it on your laptop and now want to know how to send it to Testers or to the Store.

Environment

This blog post is covering building via Azure Devops but there are many other options, like Github.

In Azure Devops you can setup your own build agent by following this guide: https://learn.microsoft.com/sv-se/azure/devops/pipelines/agents/osx-agent?view=azure-devops

You can also use a hosted agent. Here are some different alternatives: https://learn.microsoft.com/sv-se/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

For building a Maui app for iOS you need a Mac. In this build pipeline 'macos-14' is used.

pool:
  vmImage: 'macos-14'

Global settings

To share settings between build pipelines or to just abstract away the settings you can use Library in Azure Devops.

In this example the variable group "Svetto" is used.

You define the variable group in your build pipeline like this:

variables:
  - group: Svetto

This is all the setting which is used to build both for iOS and Android, the name of the files is also specified as variables.

For signing the Android app:

AndroidKeyStoreAlias
AndroidKeyStoreFile
AndroidKeyStorePassword

For signing the iOS app:

AppleCertificate
AppleSigningIdentity
iOSCertPassword
ProvisioningProfile

The version (major.minor)

ApplicationDisplayVersion

The files needed to sign the app are:

.p12 which is the Apple Certificate
.mobileprovision which is the Apple Provision Profile
.keystore which is used to sign the Android app

Version number

A version number contains of three digits {major.minor.build}.

The major and the minor are specified in the project file:
<ApplicationDisplayVersion>2.0</ApplicationDisplayVersion>

To make the build number unique build/runner in devops is often used but it's also possible to use the date like this:

<ApplicationVersion>$([System.DateTime]::Now.ToString('yyyyMMddHH'))</ApplicationVersion>

Preparing

Select Xcode version

If you want to use a specific version of Xcode you can use this inline script, here it's specified that Xcode 16 should be used.

- task: CmdLine@2
  displayName: 'Selects a specific version of Xcode'
  inputs:
    script: 'sudo xcode-select -switch /Applications/Xcode_16.app/Contents/Developer'

Install latest .net MAUI

- task: CmdLine@2
  displayName: 'Install Latest .NET MAUI Workload '
  inputs:
    script: 'dotnet workload install maui'

You can adjust the --source to target specific versions of .NET MAUI workloads, and you can learn more about changing sources in the .NET MAUI Wiki (https://github.com/dotnet/maui/wiki/#install-net-6-with-net-maui).

Install apple certificate and provision profile

For iOS, you need a signing certificate and a provisioning profile. This guide walks you through how to obtain these files:(https://learn.microsoft.com/sv-se/azure/devops/pipelines/apps/mobile/app-signing?view=azure-devops&tabs=yaml#apple)

- task: InstallAppleCertificate@2
  inputs:
    certSecureFile: '$(AppleCertificate)'
    certPwd: '$(iOSCertPassword)'
    keychain: 'temp'

- task: InstallAppleProvisioningProfile@1
  displayName: 'Install app store provisioning profile'
  inputs:
    provisioningProfileLocation: 'secureFiles'
    provProfileSecureFile: '$(ProvisioningProfile)'

Add Android keystore key

For Android, you need keystore file and value of keystore password and keystore alias. This guide walks you through how to obtain these files: (https://learn.microsoft.com/sv-se/azure/devops/pipelines/apps/mobile/app-signing?view=azure-devops&tabs=yaml#sign-your-android-app)

- task: DownloadSecureFile@1
  name: keystore
  inputs:
    secureFile: '$(AndroidKeyStoreFile)'

Building

Build iOS

In .NET 8, the dotnet publish command defaults to the Release configuration. Therefore, the build configuration can be omitted from the command line. In addition, the dotnet publish command also defaults to the ios-arm64 RuntimeIdentifier. The RuntimeIdentifier can also be omitted from the command line.

.NET MAUI apps produce executables for each Target Framework—these are native app packages with all dependencies/resources bundled in.

- task: DotNetCoreCLI@2
  displayName: 'dotnet publish iOS'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '**/Svetto.sln'
    arguments: '-f net8.0-ios -c Release -p:ApplicationDisplayVersion=$(ApplicationDisplayVersion) -p:ArchiveOnBuild=true -p:EnableAssemblyILStripping=false -p:RuntimeIdentifier=ios-arm64'
    zipAfterPublish: false

Copy ipa to stage folder

- task: CopyFiles@2
  displayName: 'Copy iOS artifact'
  inputs:
    Contents: '**/*.ipa'
    TargetFolder: '$(Build.ArtifactStagingDirectory)'
    CleanTargetFolder: true
    OverWrite: true
    flattenFolders: true

Build Android

- task: DotNetCoreCLI@2
  displayName: 'dotnet publish android'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '**/Svetto.sln'
    arguments: '-f net8.0-android -c Release -p:ApplicationDisplayVersion=$(ApplicationDisplayVersion) -p:AndroidKeyStore=true -p:AndroidSigningKeyStore=$(keystore.secureFilePath) -p:AndroidSigningKeyPass=$(AndroidKeyStorePassword) -p:AndroidSigningStorePass=$(AndroidKeyStorePassword) -p:AndroidSigningKeyAlias=$(AndroidKeyStoreAlias)'
    zipAfterPublish: false

Copy aab to stage folder

- task: CopyFiles@2
  displayName: 'Copy android artifact'
  inputs:
    Contents: |
      **/*Signed.aab
    TargetFolder: '$(Build.ArtifactStagingDirectory)'
    OverWrite: true
    flattenFolders: true

Publishing

Publish staged files, connect them to the build task

If you want to attach the ipa/abb to the build task you can use this task to publish the artifact to the task, as a zipfile called 'drop'.

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: drop'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'

Publish to Apple TestFlight

Prerequisites

In order to automate the release of app updates to the App Store, you need to have manually released at least one version of the app beforehand.
The tasks install and use fastlane tools. fastlane requires Ruby 2.0.0 or above and recommends having the latest Xcode command line tools installed on the MacOS computer.

Read more about the extension here:
https://github.com/microsoft/app-store-vsts-extension/tree/master

- task: AppStoreRelease@1
  displayName: 'Publish to Test Flight'
  inputs:
    serviceEndpoint: 'AppleTestFlight'
    releaseTrack: 'TestFlight'
    appIdentifier: 'xxxxxxxx'
    appType: 'iOS'
    appSpecificId: 'xxxxxxxx'
    shouldSkipWaitingForProcessing: true
    isTwoFactorAuth: true

Publish to Google Play Console

To publish a .NET MAUI Android app for Google Play distribution requires that your app package format is AAB, which is the default package format for release builds. To verify that your app's package format is set correctly:

In Visual Studio's Solution Explorer right-click on your .NET MAUI app project and select Properties. Then, navigate to the Android > Options tab and ensure that the value of the Release field is set to bundle:

The first time an AAB is submitted to Google Play, it must be manually uploaded through the Google Play Console. This enables Google Play to match the signature of the key on all future bundles to the original key used for the first version of the app. In order to upload the app through the Google Play Console, it must first be built and signed in Visual Studio.

- task: GooglePlayRelease@4
  inputs:
    serviceConnection: 'GooglePlayConsole'
    applicationId: 'xxxxxxxx'
    action: 'SingleBundle'
    bundleFile: '$(build.artifactstagingdirectory)/*.aab'
    track: 'internal'
    isDraftRelease: true

Conclusion

Next year AppCenter will be deprecated so MAUI developers need to find other ways to share their app to testers and users. Publishing them to the test stores (TestFlight/Google Play Console) is the recommended way.

I hope this guide will help you. Even if it doesn't cover all details about certificates, configuration and setup in the stores.

Good luck!

Connecting Azure SQL-Database and App Service using system-assigned identity

Emil Pettersson — Fri, 31 May 2024 08:44:01 +0000

I had built an application using an Azure SQL-Database for storage and I wanted to upload it to Azure as an App Service to run it in the cloud. I had assumed it was going to be like connecting an App Service to BlobStorage or TableStorage in Azure, but it turns out it is a bit different for a SQL-Database. Primarily how the system-assigned identity is used to authenticate. After figuring out how to get it to work, I decided to write the short guide I myself would have wanted to find when I started googling on why I couldn’t connect to my database.

Creating the database

To create a database we will first have to create a SQL-Server resource to host the database. When creating we will use the Authentication Method “Use Microsoft Entra-only authentication”, and set ourselves as Admin for the server.

To allow other azure resources to access our new server we have to go to Security > Networking. There we select to give public network access to “Selected networks”.

We then add our own IP Address to the Firewall Rules since we will want to test connecting to our SQL-Server locally. It is also needed to log in to our SQL-database in Azure Portal later.

In the special case that your App Service and SQL-Server will be in different Azure Subscriptions. you will have to check “Allow Azure services and resources to access this server” under Exceptions. Note that this is a big risk since it opens up your SQL-Server to EVERY Azure service and resource, even those belonging to other subscriptions and other Azure users, customers and organizations. To be secure there are solutions using virtual networks and configuring Private Access. But this is not included in the scope of this tutorial, so we will accept the shortcut of allowing this exception. This can also be a tool for troubleshooting if you suspect you have problems with connecting to the SQL-Server.

Next step is to create the SQL Database, so go to Overview and select Create Database. In the tutorial we assume the database is only for learning purposes so we want to keep costs as low as possible. So for Compute + Storage we select “Configure database” and slide the vCores and Data max Size to their lowest settings.

Then for Backup storage redundancy we select “Locally-redundant backup storage”, the weakest kind.

When the resource has been created, go to Query editor (preview) and log in with Microsoft Entra authentication. If it does not work make sure you have whitelisted your own IP.

We then create a new table by running:



CREATE TABLE animals (id INT PRIMARY KEY, name VARCHAR(100));

This creates a very simple table called “animals” where we can list different kinds of animals. Add a few animals:



INSERT INTO [dbo].[animals] (id, name) VALUES (1, 'Lion');
INSERT INTO [dbo].[animals] (id, name) VALUES (2, 'Fox');
INSERT INTO [dbo].[animals] (id, name) VALUES (3, 'Cat');

You can then see your animals by running:



SELECT * FROM [dbo].[animals]

And you can count them by running:



SELECT COUNT(*) FROM [dbo].[animals]

Creating the code

We will now build a small C# program that connects to the SQL Database and runs the SQL-command for counting the animals. Create a new ASP.NET Core Empty project in Visual Studio.

This gives you a project with only a Program.cs with minimal amount of code. We will be able to write our whole program here. First thing to do is to install the NuGet Microsoft.Data.SqlClient;

We then create a minimal api to connect to our SQL-database and execute the command to count our animals, and delete the auto-generated code we won’t need. This leaves us with a Program.cs like:

Then we go to Overview in your database-resource and select “See connection strings” under Connect to application. Take the connection string for “ADO.NET (Microsoft Entra passwordless authentication)” and put it into your appsettings of the newly created project.

For me, where both SQL-Server and SQL-database are named “charlie-foxtrot-sql” it will look like:

(Note1: Treat your Connection Strings as secrets, the resources my connection string are referencing will have been deleted when this tutorial is published)
(Note2: In the connection string you can see that we set authentication to be “Active Directory Default”. Entra ID used to be called Active Directory. This setting means that we can authenticate with our account assigned in visual studio when we run locally, and with managed identities given certain roles when we run it as an App Service)

Then run your program and make a call to the /test-db endpoint. You can do this using swagger, your favorite testing tool (e.g Postman or Insomnia) or your web browser.

If you get an error “Globalization Invariant Mode is not supported” you’ll have to unload your project and remove

from your project file (or set it to false). Then load the project again.

If you get trouble with authentication, make sure you are using the same account for Azure Service Authentication in Visual Studio as you used to create the SQL-Server and database in Azure, and that the account has the proper IAM-role to access them. You can check this under Access Control (IAM) for your SQL-Server.

You should then get a response that corresponds to the value you get by running the count-command directly in the Query Editor in Azure Portal.

Now we will create an App Service to run this code in Azure.

Creating the App Service

Create a repository for the code in Github (or Azure DevOps or Bitbucket), and push your code, except for your appsettings, to it. We will deploy the code to our App Service from there.

In Azure Portal, go to App Services and press Create > Web App and create your App Service. Then we have to connect our App Service to Github so that it knows which code to deploy. We get two options for Github Action workflow to authenticate to our App Service when deploying; the default “User-assigned identity” and “Basic authentication”. If you have sufficient permissions to use “User-assigned identity” that is recommended. You need to be able to assign role-based access to the identity. If you lack these permissions you can instead use “Basic authentication”.

If you use “Basic authentication” you first have to go to Settings > Configuration in your App Service and put SCM Basic Auth Publishing Credentials to On and press Save. If you use “User-assigned identity” you don’t have to do this.

Then go to Deployment > Deployment Center and choose your Source for the code. Choose your way of authentication and press Save.

Make sure that it builds. Then go to Settings > Environment variables and add your connection string.

Make sure you click Apply once after adding this applications setting, then once again to confirm you are done editing your application settings in general.

Note the naming of the Application Setting and compare it to how it is in your local appsettings.json. This is how Azure deals with hierarchy in JSON;



"Level1": {
    "Level2": {
        "Level3:": "Value",
    }
},

would in the Azure App Service application settings become a setting with the name
"Level1:Level2:Level3" and the value “Value”.

Then go to Settings > Identity and set Status to On for System Assigned identity. Remember to save.

For most resources in Azure you would then give a role to this managed identity that let the App Service access it. This is called Role-Based Access Control (RBAC). But for Azure SQL Database we have to do it in a different way; We want to add our App Service as a user to the database and then give the user read and write permissions. So go to the Query Editor in your database and run



CREATE USER [<appservice>] FROM EXTERNAL PROVIDER;

where is the name of your App Service. Then run both



ALTER ROLE db_datareader ADD MEMBER [<appservice>];
ALTER ROLE db_datawriter ADD MEMBER [<appservice>];

to give the new role proper permissions.
(Note: In our code we are only reading from the database, but the common case is to want both read and write permissions.)

Now you can go to your App Service Overview where you will see which “Default Domain” your app has

go to the test-endpoint of this domain. My App Service is called charlie-foxtrot-sql so for me it is https://charlie-foxtrot-sql.azurewebsites.net/test-db. This should now show you the number of animals in your database.

Expedition logbook: Journey into the world of stable diffusion

Malcolm Eriksson — Fri, 03 Nov 2023 16:56:18 +0000

This will be a summarization on a high level of my experience and some related learnings from a hobby project trying to generate images with stable diffusion in the cloud.

Intro: Preface & vision

I had previously played around a bit with generating images using Dall-E 2 and similar solutions using some free credits. My problem was this; I kept forgetting to use my free credits and thus not getting them renewed. And when I finally wanted to generate an image I ran out of credits after a few tries.

The initial idea was simply to set up a scheduled generation of images that would make sure to consume all of my free points in the various offerings available and to then automatically push them to some kind of screen or digital photo frame.
The primary images I wanted to generate would be

As I eventually got started on the project I realised that the free offerings of the various resources have been cancelled. Thus I decided to set up my own image generation using open source solutions with the goal of having it live in the cloud to make it simpler to share and to avoid needing specific local resources.

Chapter 1: Generating images

As stated in the preface; as a lazy (efficient) developer I started my journey of finding a open source implementation of one of the Image generation models, and ended up with Stabilityi-AI's Stable Diffusion setup

After fighting my Windows installation getting all the python packages, conda installations etc. running I was finally able to generate an image.

Well, that did not go great. So I kept trying making more images; altering the prompt and the configurations. All of my attempts turned out similar; but why?

To keep runtime low I tried generating images of 256x256 ratio. I tried altering the prompt, the steps (iterations) as well as CFG scale (multiplier of how much the prompt input should dictate the outcome).
But it just turned into different noise; still no images that looked anything like the prompt.

To ensure that my setup wasn't faulty I took a step back and tried some reference prompts and seeds and managed to generate images that looked like the prompt rather than random colourful noise.

1.1 What went wrong?

My primary problem was simply that I did not understand how diffusion models work well and what might be a good expected prompt (including config) to generate images.

A simplified explanation is that (image) diffusion models are trained on a specific set of images in order to learn how to remove noise from an image in order to get closer to the original image it trained on.
This means that a model can not generate outputs (in a controlled fashion) it's not trained on. This includes the ratio as it's part of the noise removal. Other ratios might still end up looking good; but it might also cause unexpected corruptions
Thus in hindsight I realise that I had to low of a CFG-scale, to few iterations and an unexpected ratio as this model was trained on 528x528 images.

Moving on from this I got back to trying to generate my original prompts; but now with new ratios, iterations and CFG scale. So what was my prompt? Variations of Fox working with a laptop in an office drinking a cup of coffee.With various alterations of phrasing and omitting details. Why this prompt? Well it's simply a reference to my the company I work at Charlie Foxtrot; a software development consultancy with foxes in our branding.

I ended up with images to containing all the contents of the prompt; even if very disjoint and not a combined picture.

I thought this might simply be to hard a prompt for this model. So I changed the prompt again; keeping the fox but altering the rest of the prompt to be a more natural setting for the fox.

Much better. After this I got back to trying various versions of the original prompt with smaller alterations for each prompt.

1.2 The end

Then my laptop blue screened. I left it a short while to cool down and then started it up again; but now for some reason my python and conda setup refused to work.
During my setup I obviously fucked up some step which left it in a state that a fresh start of the system were lacking some kind of path var or similar.

Part of the problem is of course that I did not pay enough attention when installing my dependencies and setting up my paths etc.

To avoid this happening again, to be able to share the setup with friends and colleagues as well as my future plan of getting it set up in a cloud solution I decided to get setup in a Docker image instead. In addition to this I decided to look for a more feature complete solution; preferably with an already prepared docker setup.

1.3: Giving it another shot

As mentioned I wanted a more feature complete solution with an existing docker setup; and I found it InvokeAI.

While this probably relates more to my setup and experience; setting this up was a lot smoother compared to my initial setup, I simply had to build the image according to instrucitons and run it. I suddenly was making images with outcomes that looked like my prompt with a provided GUI. When my laptop inevitable blue screened again from being pushed past its limit I recovered in seconds.

But the images took longer to generate than expected and the gpu was hardly working even when I tried to pass through GPU resources. Did this have something to do with InvokeAI, the model or my setup? After verifying the requirements I simplified the setup by using invokeai's simple installer version; removing potential issues with Docker (running via WSL).

I instantly got better performance and thus could figure out that something was not working as intended. With this knowledge I could debug and resolve the issue with the gpu passthrough and get back to my docker setup.

After this I simply dialed in the prompt configuration to find a good balance between output and runtime by testing various combinations of prompts, cfg and steps.
When this was done I tried all of the recommended models from the invokeai installer and tried a few iterations of each to find the model(s) that I thought generated the best images for my given prompt. For some models I had to play around a bit with new baselines for cfg/steps to get images I thought looked good.

I could now generate images in bulk where some of them actually looked as intended.

1.4 Learnings

Some simple learnings from this:

Always work with a recoverable / reproducable environment.
Understand the tools (models) better when using them. Even when playing around you need to play around in a mindful manner.
When experimenting always sanity check with a known input/output as soon as possible. It might seem more fun to have the first outcome to be what you want; but you probably want to verify your setup first.
Image generation is just structured randomization. You will need to generate a lot of images and sort through them either manually or with image analysis to provide outcomes you're happy with. Finetuning input for your model can take a lot of iterations.
Schedule the time for your input/model testing better than I did; generating images is a resource hog and it will make using the device for anything else at the same time a pain. Alternatively get it set up on a secondary machine asap when you know you have working setup.

Chapter 2: To the cloud

As I now had a dockerized soloutin that I liked

After finding models and configurations that performed to my liking on my laptop it was now time to get into the cloud. This is for multiple reasons. Amongst others it would allow for scaling above physical local devices, would not bog down said local devices during runtime and in general be better for sharing and demo purposes.

So what's on the shopping list:

Container Hosting (potential autoscaling)
Gpu resources for the container
A storage volume to make outputs, models and invokeai setup stable between instances.
Keeping costs down

To simplify the decisions I limited myself to the Azure platform as my company already had an existing setup for it.

2.1 Container hosting

So lets start with getting the container into the cloud; that's the minimum working solution. Getting something autoscaling would simplify maintenance and in general be resource efficient.

Starting with simple azure container apps would allow for azure handled autoscaling and some free resources monthly; which would be nice for a project of this scale. Sadly the maximum memory of azure container apps of 4Gi is way below the required specs for invokeai of 12GB.
I never tried how it would perform on these resources and there is the possibility to request a manual increase of memory limit.
So this is something that could be explored in the future.

Secondly we could explore azures kubernetes setup aks.

Sadly on a free tier there is no autoscaling which made me opt out from this option. However neither the standard tier cluster nor the nodes are that expensive and there are a lot of GPU supported options which might be worth trying out in another iteration in the future.

However I opted for a regular container instance. In theory there should also be cost effective K80 GPU's that could be used; however it's a preview feature which is not intended for production use and needs to be manually approved by Azure prior to usage.
Our request got rejected; hence we are running without a GPU in the cloud.

The pricing for our needs is low; 1 cpu and 12 GB memory which totals to ~$2 per day of uptime; and with mindful shutdown of the service we would of course only pay for actually active resources. And even without a gpu we can produce images in a somewhat acceptable time.

In addition to all this we will of course need a registry. This costs $0.167 / day on the basic tier.

2.2 Cloud storage volume

Setting up a volume mount for the container seemed pretty simple. All I needed was to set up a file share and mount it to my container instance.

Depending on your setup you should only need ~12GB for your models as well as extra storage for caching between steps of the image generation + generated output storage. I could not see storage going above ~25GB with some mindful cleaning of unused data. At this point pretty much any storage option would be cheap; I went for a transaction optimized setup at $0.06 per used GiB and month for the storage + $0.015 per 10000 read/writes. At this price point we should not need to worry at all.

The first few burst tries of starting one docker container; generating one image and then killing everything this worked fine.

I then tried sharing the volume between two different container instances and generated multiple images on both instances.
Sharing a volume worked fine and I thought that was it; I generated a few images each for a few hours straight.

Happy with the performance and results I closed them down for the day.

Next day I sanity checked my cost analysis and realized that for some reason I had an unexpectedly huge storage cost. Turns out that this also induces a network fee for the file transfers; while not immensly large this costed about 90% of the storage cost and even 90% of the entire project at this scale of use.

Thus I decided to cut it from the project to keep costs down.
The only implication from this is that I need to reinitiate the invokeai setup for every container on startup; re-install any models I want to use and ensure to download any outputs I want to keep before shutting it down. For this type of hobby project that's perfectly fine.

When the week was up it was apparent what had happened; azure gave me the finger.

2.3 Learnings

So what did I learn this time

Start any request for potential resource needs way ahead of time; there might be a lot of forth and back and you might eventually get rejected.
Read pricing details more closely so you understand potential extra costs in your calculations.
When trying out things do it in iterations and inspect the outcomes after use vigilantly. Had I paid more attention the first few days the spike would have never occured. This time the cost was just high relative to an extremely low baseline; but in any project costs can run amok if you don't pay attention.

Outro: The outcome and next steps

After all of this I opted to let the iteration 0 frame to be a simple solution, we bought a photo frame with a proprietary application for sharing images.

There are a lot of places to take this project in a second iteration; building a custom photo frame with functionality for automatically pushing new images generated to it, training / extending a model to use the companys mascot and logos rather than generic foxes, a different cloud hosted solution with the same or different provider, using any of the many other features invokeai supports like multistep workflows, or simply replacing invokeai with a different solution. Or maybe this is the final iteration of the project?

Rpi 4 meets Flutter and Rust

AdaShoelace — Fri, 06 Nov 2020 16:05:05 +0000

At Charlie foxtrot we aim to stay relevant and keep up with new and exciting tech. So how do we do that? We experiment! This time we wanted to capitalize on our in-house competence in Flutter and Rust. Our Flutter expert, Jonathan, had read about Darts foreign function interface (ffi) capabilities and wanted to try it in conjunction with Rust. He asked me if I thought if we could build something that would run on a Raspberry pi (rpi). Without having a clue, I said “YES!” and got to work. The resulting code can be found here.

After some debating, we landed in an idea where we would have a Flutter interface that let the user take a photo via the rpi camera module and then display the image in the app. This is a simple app but considering that Flutter don’t have official support for the rpi, this could be quite the challenge. We also decided to build a small (read: very small) library in Rust to communicate with the camera.

This project has quite a few requirements:

Rust
Flutter
flutter-pi
rpi 4 4gb ram
an rpi camera module
a touch screen, we went with the Hyperpixel 4.0.

Setup

First thing to do is to set up the Raspberry Pi. In the README of the flutter-pi repo it's mentioned that flutter-pi should run without Xserver. Therefore I went with a headless version of RaspberryOS. Once you've flashed the os to a ssd card, follow the instructions in the flutter-pi README and install drivers for your touch screen of choice. Be sure to clone the engine-binaries branch of flutter-pi and place the files in the correct directories aswell.

On the software side we first want to set up Flutter on our laptop. If you are using a linux distro you may have the option to install and run Flutter via snap. I found that this leads to several issues. Because of that I cloned the Flutter repo from Github and installed it manually. Where you install Flutter doesn’t really matter as long as you put /flutter/bin in your $PATH.

Next you want to make sure you are using a Flutter version that’s compatible with flutter-pi. At the time of writing this the compatible version is 1.22.2. You can change Flutter-version by checking out the corresponding branch. You can now test your installation by running flutter create <name> followed by flutter build bundle. If it builds without errors, everything should be correct.

We will also need to install Rust so that we can build the small library that lets us talk to the camera. Do this by going over to the Rust website and follow the instructions. Once Rust is installed we need to add the armv7 target for the rust compiler in order for us to cross compile the code. Do this with: rustup component add rust-std-armv7-unknown-linux-gnueabihf. Now create a Rust project using cargo new –-lib ffi-test. The last part (ffi-test) is the name of the project and doesn’t matter, choose a name to your liking. As I mentioned earlier we need to cross compile the code since it’s going to run on the rpi and not the machine we are writing the code on. Create a directory called .cargo and place a file in there named config.toml. In that file place the following:

[build] 
target = "armv7-unknown-linux-gnueabihf"  

[target.armv7-unknown-linux-gnueabihf] 
linker = "arm-linux-gnueabihf-gcc"

The last part after “linker =” is the gcc linker for arm. Install this if you don’t already have it on your machine. Google the correct name of the package for your distro. The name displayed here is the package name on Manjaro.
We also need to declare, inside the Cargo.toml, that we want to build a dynamically linked library (.so) like this:

[lib]
name = "ffi_test"
crate-type = ["cdylib"] #dynamic library

The next logical step would be to build a small "Hello world" library in Rust and consume it from Dart/Flutter. Lets create a function that returns a hard coded string when called.

#[no_mangle]
pub extern fn string_from_rust() -> *const c_char {
    let s = CString::new("Hello from Rust!").unwrap(); // Create a CString from "Hello world"
    let p = s.as_ptr(); // Get a pointer to the underlaying memory for s
    std::mem::forget(s); // Give up the responsibility of cleaning up/freeing s
    p
}

Since Dart's ffi mechanisms utilizes the C ABI, we need to annotate the Rust function with #[no_mangle] in order to preserve the function signatures symbol upon compilation. Since we are compiling this into a .so we need to mark the function as extern so that it can be linked as any other library. The functions return type is *const c_char, which is a raw pointer, basically a raw memory adress. We have to do this since Rust's types are not available in Dart. When we send data over to Dart, Rust can no longer take responsibility for cleaning up that data. Rust no longer have the required information regarding that data's usage and could therefore free the data at an inappropriate time. To build this just run cargo b

Now we want to consume the Rust function from the Dart side of the project. To do that we need the following dependencies:

import 'dart:ffi' as ffi;
import 'dart:ffi';
import 'dart:typed_data';
import 'package:flutter/material.dart';
import 'package:ffi/ffi.dart';

Now we can load and use the Rust library:

typedef NativeRustStringFromRustFunction = ffi.Pointer<Utf8> Function();

class _MyHomePageState extends State<MyHomePage> {
  _MyHomePageState() {
      this._dl =
          ffi.DynamicLibrary.open("libffi_test.so");
      this._string_from_rust_ffi =
          _dl.lookupFunction<NativeRustStringFromRustFunction, NativeRustStringFromRustFunction>(
          "string_from_rust");
  }
  int _counter = 0;
  String _message = "Photos taken: 0";
  ffi.DynamicLibrary _dl;
    NativeRustStringFromRustFunction _string_from_rust_ffi;
  void _updateMessage {
    setState(() {
      _counter++;
      var foreignMessage = ffi.Utf8.fromUtf8(_string_from_rust_ffi());
      _message = "'$foreignMessage': '$_counter'";
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(_message),
            Text(
              '$_counter',
              style: Theme.of(context).textTheme.headline4,
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _updateMessage,
        tooltip: 'Increment',
        child: Icon(Icons.add),
      ),
    );
  }
}

We first load the .so library we compiled from Rust, we look up the function and store it in the state. We then write a small helper function that will call the _string_from_rust_ffi-function. Then we have to encode the string returned from Rust to UTF8 using Utf8.fromUtf8().

In order to test this enable desktop support with:

$ flutter channel dev
$ flutter upgrade
$ flutter config --enable-linux-desktop

Check the config with flutter devices and see that your os shows up. Then to run the app run flutter run -d linux. You should something like this:

Moving over to the Rpi

The main goal in this post is to get Flutter running on the rpi while consuming a Rust library. So far we haven't run anything on the rpi. Lets fix that!

Flutter does not have any official support for rpi, that's why we need to use the aforementioned flutter-pi. If you have followed the instructions in the repo for flutter-pi you should have it installed on the rpi. It would be ideal if we could just take the code we built earlier and throw it on to the rpi and run it. Unfortunately that's not the case. The engine-binariesbranch from the flutter-pi repo needs to be cloned to the "host" (the computer you are coding on) in order for us to build and compile the Flutter app. Once you've done that go through the build steps in the README of flutter-pi. To save myself time I've saved these as bash scripts. These are listed below.

build_snapshot.sh

#!/bin/bash
~/flutter/bin/cache/dart-sdk/bin/dart \
    ~/flutter/bin/cache/dart-sdk/bin/snapshots/frontend_server.dart.snapshot \
    --sdk-root ~/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk_product \
    --target=flutter \
    --aot \
    --tfa \
    -Ddart.vm.product=true \
    --packages .packages \
    --output-dill build/kernel_snapshot.dill \
    --verbose \
    --depfile build/kernel_snapshot.d \
    package:hello_world/main.dart

build_so.sh

#!/bin/sh
~/engine-binaries/gen_snapshot_linux_x64 \
  --causal_async_stacks \
  --deterministic \
  --snapshot_kind=app-aot-elf \
  --elf=build/app.so \
  --strip \
  --sim_use_hardfp \
  --no-use-integer-division \
  build/kernel_snapshot.dill

build.sh

#!/bin/bash
flutter clean && \
    flutter build bundle --release && \
    ./build_snapshot.sh && \
    ./build_so.sh \

To build for flutter-pi now run the build.shscript.
Upload the files to the rpi using the upload.sh script:

#!/bin/bash
rsync -a --info=progress2 ./build/flutter_assets/ pi@raspberrypi:/home/pi/flutter_hello_world_assets && \
    scp ./build/app.so pi@raspberrypi:/home/pi/flutter_hello_world_assets/app.so

To run the application ssh into the rpi and run flutter-pi flutter_hello_world_assets. If all goes well you should see the "Hello world" app from Flutter and once you press the button the text changes to "Hello from rust.

Incorporating the camera

Now that we know how to compile and run Rust and Flutter in conjunction on the rpi we want to be able to control the camera from our app. There might be libraries for controlling the camera written in Dart but since we want to use Rust together with Dart, we're going to do it from Rust.

I've chosen a library called rascam. It's a small library built on top of mmal-sys. We can add rascam to our Rust project by adding this to the Cargo.toml file:

[dependencies]
rascam = "*"

Now lets take a look at the code for taking an image.

// ffi-test/src/lib.rs
use std::ffi::CString;
use std::os::raw::c_char;
use rascam::*;

// ...

#[repr(C)]
pub struct ImageBuffer {
    img_ptr: *mut u8,
    len: u32,
}

#[no_mangle]
pub extern fn take_photo() -> *mut ImageBuffer {
    let info = info().unwrap();
    if info.cameras.len() < 1 {
        println!("Found 0 cameras. Exiting");
        // note that this does not run destructors
        ::std::process::exit(1);
    }
    println!("{}", info);
    let mut camera = SimpleCamera::new(info.cameras[0].clone()).unwrap();
    camera.activate().unwrap();
    let mut bytes = camera.take_one().unwrap();
    let len = bytes.len() as u32;
    let ret = bytes.as_mut_ptr();
    std::mem::forget(bytes);
    let mut ib = Box::new(ImageBuffer { img_ptr: ret, len});
    let ret = Box::into_raw(ib);
    std::mem::forget(ret);
    ret
}

There is a lot going on here. If youve never ventures in to the unforgiving territory of low level Rust, this might look a little daunting. But fear not! I will explain it all, piece by piece.

On a high level this code solves two problems, take a photo and send it to whoever is calling the function. We will start with the second problem, sending the photo to the caller. Since it's just two values we we want to pass on, one might be tempted to utilize a tuple like (*mut u8, u32). This is however not a viable option since the memory layout of a tuple in Rust is not guaranteed. So we went with a simple struct. The #[repr(C)] tells the compiler to treat the struct as a C struct. This means that the fields of the struct will be aligned the same way in memory as we declared them in the code. We need this because the C ABI being the common grounds for communication between Rust and Dart.

We start with getting the necessary info for the camera. We use this to check if there even is a camera to begin with. If there isn't we just close the process.

  let info = info().unwrap();
  if info.cameras.len() < 1 {
      println!("Found 0 cameras. Exiting");
      // note that this does not run destructors
      ::std::process::exit(1);
  }
  println!("{}", info);

We then instatiate the camera using the info and activate it.

 let mut camera = SimpleCamera::new(info.cameras[0].clone()).unwrap();
 camera.activate().unwrap();

We take a photo, get the number of bytes, get a pointer to the byte array and tell the compiler to forget about the byte array. Just as with the string earlier this array will be cleaned up by Dart.

  let mut bytes = camera.take_one().unwrap();
  let len = bytes.len() as u32;
  let ret = bytes.as_mut_ptr();
  std::mem::forget(bytes);

Next we create an ImageBuffer and place it on the heap by wrapping it in a Box. If we didn't place it on the heap and still returned a pointer to it, the ImageBuffer would be popped of the stack once the function call returns. That would make it possible for the memory to get overwritten while Dart uses the pointer. We don't want that!

  let mut ib = Box::new(ImageBuffer { img_ptr: ret, len});
  let ret = Box::into_raw(ib);
  std::mem::forget(ret);
  ret

With that everything is well and done on the Rust side but how do we actually use this in Dart?
First will need to declare a type to represent the ImageBuffer in Dart.

class ImageBuffer extends Struct {
  Pointer<Uint8> img_ptr; 

  @Uint32()
  int len;
}

then we need a type for the actual function providing us the ImageBuffer and we need to look it up in the library:

  typedef NativeRustTakePhotoFunction = ffi.Pointer<ImageBuffer> Function();
  // inside MyHomePageState
  _MyHomePageState() {
      this._dl =
          ffi.DynamicLibrary.open("/home/pi/target/debug/libffi_test.so");
      this._take_photo_ffi =
          _dl.lookupFunction<NativeRustTakePhotoFunction, NativeRustTakePhotoFunction>(
          "take_photo");
  }

We then declare a function to be called on button press:

  void _takePhoto() {
    setState(() {
      _imageBuffer = _take_photo_ffi().ref; // _imageBuffer is a member of our state
      _counter++;
      _message = "Photos taken: '$_counter'";
    });
  }

Once all that is done we can register the function with the button:

  // Inside a widget array in a scaffold
  if (_imageBuffer != null) 
    Image.memory(_imageBuffer.img_ptr.asTypedList(_imageBuffer.len)),

Unfortunately I couldn't build the Rust library on my Linux machine because of mmal-sys. There might be a solution for this. But to build this I:

pushed the Rust library code to github
built the flutter bundle on the host, using the scripts
uploaded it to the rpi and ran it. It then gave me this:

Now you should hopefully have a Flutter app running on a rpi!

Takeaways

For me the main takeaway is that our approach here might actually be a viable option for resource scarce devices. The possibility of writing a beautiful, interactive application in Flutter and have run in a car infotainment system for example is quite exciting. The possibility to handle the low level stuff with languages such as Rust and C, really opens up this domain for Flutter.
I at least know that I am waiting for official rpi support for Flutter so that I can build something more robust!

Drawing arrows in Flutter, or anything else, based on widget positions

Jonathan Böcker — Fri, 07 Aug 2020 07:19:32 +0000

I recently made a Flutter package called widget_arrows which caught some attention on the FlutterDev subreddit.
I thought I'd take a minute to explain how it works, since the technique can be applied to other things, such as a tutorial hole overlay.

Schwusch / widget_arrows

Draw arrows between widgets in Flutter

widget_arrows

Draw arrows between widgets

View on GitHub

Using it is fairly straight forward, wrap the Widgets you want to draw arrows between with an ArrowElement and wrap both with an ArrowContainer higher up in the Widget hierarchy .

In this case I wrap the whole MaterialApp in an ArrowContainer in order to display the arrows while dragging an element:

The core Widget is the ArrowContainer, which is the one drawing the arrows on a canvas layered on top of the child, using a Stack.
Here's an example:



ArrowContainer(
  child: Column(
    children: [
      ArrowElement(
        id: 'top',
        targetId: 'bottom',
        child: Text('Text1'),
      ),
      ArrowElement(
        show: showArrows,
        id: 'bottom',
        child: Text('Text2'),
      ),
    ],
  ),
)

During initState(), the _ArrowElementState looks up the widget hierarchy and finds the closest _ArrowContainerState and registers itself by calling addArrow(this) on the container. The _ArrowContainerState will then trigger a repaint on the CustomPainter that draws on top of ArrowContainers child, which in this case is a Column.

How does the _ArrowElementState find the _ArrowContainerState? By calling this method on the BuildContext:



context.findAncestorStateOfType<_ArrowContainerState>();

Since ArrowContainer is a StatefulWidget, it has its associated _ArrowContainerState. It's a relatively expensive lookup, but the good news is that it is only needed once per ArrowElement.

The _ArrowContainerState has ChangeNotifier mixed in, in its class declaration. That means that it can be passed in as an argument to the CustomPainter constructor, and it will repaint every time _ArrowContainerState calls notifyListeners().

_ArrowContainerState.notifyListeners() is called every time an arrow element is added or removed, which means it is always kept in sync with all ArrowElements currently mounted in its child Widget subtree.

The _ArrowContainerState build method look like this:



@override
Widget build(BuildContext context) => Stack(
      children: [
        widget.child,
        IgnorePointer(
          child: CustomPaint(
            foregroundPainter: _ArrowPainter(
              _elements,
              Directionality.of(context),
              this,
            ),
            child: Container(),
          ),
        ),
      ],
    );

The CustomPaint is wrapped with an IgnorePointer, to let through touch events to widget.child, which is the users real UI. The _ArrowPainter is a custom class extending CustomPainter to draw the Arrows. It takes the _ArrowElementStates as a first argument, the TextDirection as the second argument (we will cover why later), and a Listenable as the third argument. The _ArrowContainerState is a Listenable, due to having ChangeNotifier mixed in.

What happens in _ArrowPainter.paint() is the real magic. It loops through all _ArrowElementStates, and calls this method:



final renderBox = arrowElementState
                    .context
                    .findRenderObject() as RenderBox;

The RenderBox is the key to finding the position of the start and end of the arrows we're about to draw.
By calling this method:



final Offset upperLeftCorner = renderBox.localToGlobal(Offset.zero);

We get an Offset of where it is relative to the top-left corner of the screen. The RenderBox also has RenderBox.size which gives us its dimensions, which is used to describe the rectangle in which the child is drawn.

This is where TextDirection comes in. The ArrowElement has a sourceAnchor and a targetAnchor argument which takes an AlignmentGeometry. This allows us to specify where in the child Widget we like the arrow to start or end from, and is Alignment.centerLeft by default. You can pass an AlignmentDirectional to make it adapt to whether the text is right-to-left or left-to-right. To adapt to the text direction, this method is called on the AlignmentGeometry object:



final Offset startPosition = sourceAnchor
  .resolve(textDirection)
  .withinRect(/*child widget rectangle*/);

From there, all that's left is to draw a Path or whatever you like on the canvas at that position.

Happy Fluttering!