DEV Community: Yang Fang

How to Setup Flutter for Building Android without Android Studio

Yang Fang — Mon, 08 Apr 2024 20:41:13 +0000

If you follow the official tutorial to install Android tools for Flutter development, it starts with downloading Android Studio. While this is the best way for beginners to get started quickly and avoid some common pitfalls, some advanced users may find it insufficient for their requirements. For example, when trying to set up a Continuous Integration environment on a headless remote server, the environment often times does not have a desktop environment and Android Studio cannot be started.

In this guide, we will walk through the steps needed to install Android build tools for Flutter to successfully build an Android application package without Android Studio.

Note: Flutter can be used to target many different platforms such as iOS, Windows, Linux, macOS, web. This guide only deals with setting up for building Android.

The steps can be roughly divided into:

Download Java and set up environment variables
Download Android SDK command line tools
Download Android SDK using Android SDK command line tools and set up environment variables
Download Flutter and set up environment variables

Download Java and set up environment variables

Note: There are many versions of Java you can download. To get proper compilation with Android, you must pick the correct version according to the project Gradle version, Android Gradle Plugin version, and Kotlin version. There are many intricacies, but you should always use an LTS version of Java, and often times the highest LTS version that has been released for at least 18 months.

If you already have Java installed, or can easily install it through the OS package manager (using APT, DNF, or RPM), you can skip this step. Make sure to point the JAVA_HOME variable to the base directory.

Get started by downloading your favorite Java distribution. There are many options to choose from. A most unique one is Eclipse OpenJ9 [download here] because it has an alternative Java library and JVM implementation. I recommend avoiding the OpenJDK.org distribution (a.k.a. reference implementation) – because they do not release security updates past the .2 release; and Oracle JDK, due to the non-free license. Other options include:

Adoptium’s build of OpenJDK - a compilation of OpenJDK by Eclipse foundation
Microsoft’s build of OpenJDK – a compilation of OpenJDK signed by Microsoft
Jetbrains Runtime – this is bundled with IntelliJ and Android Studio, but with patches focused on optimization for use within an Interactive Development Environment context – doing incremental builds, hot reload of class definition, and embedded Chromium in the runtime. I do not believe the added features are useful for building and archiving Android Applications for distribution, but it will work non-the-less.

If you have an archive file (.zip or .tar.xz), extract it with your favorite commandline program onto your preferred location. This can vary by your environment. For example:

tar -xf ibm-semeru-open-jdk_x64_windows.zip -C C:\DeveloperTools\Java or tar -xJf ibm-semeru-open-jdk_x64_linux.tar.gz -C /opt/java17/

Now set the PATH and JAVA_HOME environment variables. PATH should be appended with the path to the Java’s bin directory. JAVA_HOME should be the root (not bin) directory of the Java location. This depends on your OS. Please look that up if you do not know how to persist it. For Windows:

set PATH=%PATH%;C:\DeveloperTools\Java\bin
set JAVA_HOME=C:\DeveloperTools\Java

For Linux/macOS:

export PATH=$PATH;/opt/java17/bin
export JAVA_HOME=/opt/java17

Download Android SDK Command Line Tools

Next, we will download Android SDK’s command line tools. Go to this webpage and visit the “Command line tools only” section.

The Android SDK command line tool is the tool that allows downloading and updating other Android SDK components such as the build tool and source code through the command line. When installing without Android Studio, the command line tools must become part of the SDK installation.

Android SDK uses a specific directory structure to recognize whether its components are installed. Within the Android SDK root, each component must reside in its specificly named subfolder. The command line tools content must reside in <android sdk root>/cmdline-tools/latest.

First, pick a preferred Android SDK location, such as C:\DeveloperTools\Android\SDK. Extract the tool into a your preferred location. Then, move and rename the command line tools folder into the expected subfolder.

mkdir C:\DeveloperTools\Android\SDK
mkdir C:\DeveloperTools\Android\SDK\cmdline-tools
tar -xf commandlinetools-win-11076708_latest.zip -C C:\DeveloperTools\Android\SDK\cmdline-tools
ren C:\DeveloperTools\Android\SDK\cmdline-tools\cmdline-tools C:\DeveloperTools\Android\SDK\cmdline-tools\latest

Download Android SDK using Android SDK Command Line Tools and Set up Environment Variables

First, we will set up the environment variables. ANDROID_HOME should point to the root of the Android SDK directory. In my previous example, that would be:

set ANDROID_HOME=C:\DeveloperTools\Android\SDK

For Linux/macOS:

export ANDROID_HOME=/opt/Android/sdk

Next, we will invoke Android SDK command line tools to download the platform tools component to test our installation. Platform tools contains ADB which is needed for detecting connected Android devices. You can alternatively install Android build tools on a CI server where platform tools are not needed (replace with build-tools;30.0.3).

%ANDROID_HOME%\cmdline-tools\latest\bin\sdkmanager platform-tools

For Linux/macOS:

$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager platform-tools

This should install the platform tools, and if successful, then installation is sound.

Download Flutter and set up environment variables

This is the last step, but a lot of people probably started with this step first before looking for a tutorial. Head to Flutter SDK archive and download Flutter SDK for your OS.

Extract the SDK to a preferred location, for example

tar -xf flutter.zip -C C:\DeveloperTools\flutter or tar -xf flutter.tar.xz -C /opt/flutter

Now, set the PATH variable to the bin of flutter directory and FLUTTER_ROOT variable to the root of the flutter directory. For Windows:

set PATH=%PATH%;C:\DeveloperTools\flutter\bin
set FLUTTER_ROOT=C:\DeveloperTools\flutter

For Linux/macOS:

export PATH=$PATH;/opt/flutter/bin
export FLUTTER_ROOT=/opt/flutter

You’re not done here. You will need to execute flutter for the first time as a non-admin non-root user. You may need to restart your command line if it is using administrator mode. Now simply execute flutter --doctor. Flutter should detect Android SDK and Java using the environmental variables.

And we are done. Simply try compiling your project, or the example counter app.

An Investigation into Poor UX in IntelliJ Idea on Windows with Subsystem for Linux

Yang Fang — Fri, 10 Nov 2023 21:22:18 +0000

I recently installed IntelliJ Idea 2022.2.5 on a Windows 11 machine. I am fairly familiar with this IDE, having used it on Windows frequently from 2014 until 2018, then on Linux occasionally until 2022. And I'm also quite familiar with Android Studio, its sister product on which they share the same source code. I followed my own usual steps to install the IDE - downloading the zip, extracting to a suitable location, running the studio.bat file.

I expected the process to be a breeze, taking no more than 10 minutes. But to my surprise, after a brief setup dialog, my IDE was hanging after I click "New project" in the main project dialog. After digging deeper and deeper, what should have been 10 minutes ended up taking nearly 4 hours.

Symptoms

The bug manifests itself on a new installation of IntelliJ on a Windows 11 OS that has never run any version of IntelliJ before. WSL2 is installed with Debian 11. After clicking the "new project" button, the IDE shows a modal "Detecting JDK" window with a looping progress bar and a cancel button. The user cannot interact with any other UI element other than the modal window. But this window would not change for more than 10 minutes, appearing to not be making any progress. Clicking on the cancel button would disable the cancel button from being clicked again, but would do nothing to make this window disappear or any UI element interactable. From here, the only recourse is to kill the process in Windows Task Manager. Restarting IntelliJ would hit this issue again.

Video:

github-production-user-asset-6210df.s3.amazonaws.com

I've also tried a few other versions of IntelliJ including 2023.2, 2022.1, and 2021.2 with the same result, indicating a problem with multiple versions.

After doing some preliminary research, I found out this issue was reported by many users over the past 2 years on YouTrack (JetBrain's Bug Tracker), Reddit, StackOverflow. Here is a non-exhaustive list of reports:

https://youtrack.jetbrains.com/issue/IDEA-293604
Related but closed due to incomplete information or happened to work after Defender disabled
https://intellij-support.jetbrains.com/hc/en-us/community/posts/7125963159826
https://www.reddit.com/r/IntelliJIDEA/comments/vauqir/
https://www.reddit.com/r/IntelliJIDEA/comments/1143nq0/
https://stackoverflow.com/questions/68479755/

A few comments mentioned Windows Defender. So I tried completely disabling Windows Defender's real time protection. But this did not fix the issue for me. IDE continued to hang for 10+ minutes.

Another person on StackOverflow mentioned it having to do with Podman on WSL. Coincidentally I do use Podman on WSL2. I cannot conclusively say whether Podman is a necessary condition for triggering this bug. I would guess not given the number of people reporting this issue and the Podman not being a very common tool on a WSL2 installation.

Some of these bug reports do not have complete information (such as logs and dumps), common for reports submitted by the public. But the number of these bug reports with the same symptoms should give off something.

What surprised me though, is that JetBrains engineers could not isolate a major UX breaking issue and fix it for good after 2+ years. Many users shared their frustration on https://youtrack.jetbrains.com/issue/IDEA-293604.

UX Issues

Before we go into the cause of the slowness, we should take a step back and ask what is wrong. Sometimes what is wrong with the User Experience does not have to do with the root cause of the issue, as shown in this case.

The first principle of UX is to always show your users a "live" UI where the user is in control. Short of a complete fatal crash of the program, in no circumstance should a "hanging" UI where the user cannot do anything be shown.

Imagine if you're dialing the emergency number, but your phone decides to show you a spinning circle. Clicking cancel doesn't change anything. In the meantime, the only thing you can do is to watch the spinning circle spin. Whether the underlying issue is weak signal, phone overheat, or low battery, the UI itself is an issue.

Here is a list of improvements that can improve the UX without having to solve the root cause:

For a long blocking process (if we determined "detecting JDK" as one), an actual progress bar should be shown rather than a looping one indicating no progress
The cancel button should actually cancel the process.
Consider moving "detecting JDK" to the background if it is not blocking.
Set a timeout on "detecting JDK" if it is blocking but a partial result is satisfactory to unblock the user
Consider "detecting JDK" in steps, where a partial result is acquired initially, then the full result made available later
A warning should be shown to the user if a process is expected to take a long time. (Another example would be if the user opens a project deemed too big to load in IntelliJ, the user should be warned "you're opening a huge project with xx number of files, it may take some time" rather than faced with a non-responsive UI without knowing what the problem is)

Another UX principle this issue violates is to not make all of your users suffer for features that would only benefit some of your users. After some deeper investigation, the issue indeed has to do with IntelliJ's WSL2 support code path. WSL2 can be used to do a lot of things (perhaps I just want to play SuperTuxKart on WSL2 and nothing else). But the feature is enabled for all IntelliJ users whether the user wants to be using IntelliJ with WSL or not. There is no switch to turn off WSL2 integration in IntelliJ settings, properties, or experiments (yes I tried, including a few things that normal people wouldn't try). In that process, IntelliJ serves a poor user experience to all users with WSL2 when only some of these users are interested in using WSL2 inside IntelliJ. The improvements:

Offer an option to turn off WSL2 integration in IntelliJ settings. (It can be default ON if JetBrains wants to push WSL2 integration, though it should be a dialog asking to enable itself if you ask me)

These 2 UX principles should be kept in mind regardless of what the issue is. No slowness should lead to an unresponsive UI, and no feature with performance degradation should be forced on every user when only a portion would benefit.

Root Cause of Slowness

The following is the relevant stacktrace. This stacktrace is available either in a dump (in a few versions of IntelliJ a dump is automatically generated and sent when a hang is detected) or if you attach a performance monitoring tool into IntelliJ's JVM process.

Stack trace:
java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.OpenNtQueryDirectoryInformation0(Native Method)
java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.OpenNtQueryDirectoryInformation(WindowsNativeDispatcher.java:284)
java.base@11.0.15/sun.nio.fs.WindowsDirectoryStream.<init>(WindowsDirectoryStream.java:72)
java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(WindowsFileSystemProvider.java:344)
java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkAccess(WindowsFileSystemProvider.java:371)
java.base@11.0.15/sun.nio.fs.AbstractFileSystemProvider.exists(AbstractFileSystemProvider.java:151)
java.base@11.0.15/java.nio.file.Files.exists(Files.java:2510)
com.intellij.openapi.projectRoots.JdkUtil.checkForJdk(JdkUtil.java:87)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderBasic.scanFolder(JavaHomeFinderBasic.java:194)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderBasic.scanAll(JavaHomeFinderBasic.java:188)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderBasic.findInSpecifiedPaths(JavaHomeFinderBasic.java:87)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderBasic$$Lambda$1632/0x00000001011f7040.get(Unknown Source)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderBasic.findExistingJdks(JavaHomeFinderBasic.java:100)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderWindows$5.get(JavaHomeFinderWindows.kt:57)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderWindows$5.get(JavaHomeFinderWindows.kt:17)
com.intellij.openapi.projectRoots.impl.JavaHomeFinderBasic.findExistingJdks(JavaHomeFinderBasic.java:100)
com.intellij.openapi.projectRoots.impl.JavaHomeFinder.suggestHomePaths(JavaHomeFinder.java:73)
com.intellij.openapi.projectRoots.impl.JavaHomeFinder.suggestHomePaths(JavaHomeFinder.java:61)
com.intellij.openapi.projectRoots.impl.JavaSdkImpl.suggestHomePaths(JavaSdkImpl.java:199)
com.intellij.openapi.roots.ui.configuration.SdkDetector.detect(SdkDetector.java:149)
com.intellij.openapi.roots.ui.configuration.SdkDetector$2.run(SdkDetector.java:208)
com.intellij.openapi.progress.impl.CoreProgressManager.startTask(CoreProgressManager.java:442)
com.intellij.openapi.progress.impl.ProgressManagerImpl.startTask(ProgressManagerImpl.java:114)
com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressAsynchronously$5(CoreProgressManager.java:493)
com.intellij.openapi.progress.impl.CoreProgressManager$$Lambda$1617/0x00000001011f0040.apply(Unknown Source)
com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$3(ProgressRunner.java:252)
com.intellij.openapi.progress.impl.ProgressRunner$$Lambda$1498/0x0000000101118c40.run(Unknown Source)
com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:188)
com.intellij.openapi.progress.impl.CoreProgressManager$$Lambda$712/0x000000010087dc40.run(Unknown Source)
com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:608)
com.intellij.openapi.progress.impl.CoreProgressManager$$Lambda$509/0x000000010025e040.compute(Unknown Source)
com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:683)
com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:639)
com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:607)
com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:60)
com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:175)
com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$4(ProgressRunner.java:252)
com.intellij.openapi.progress.impl.ProgressRunner$$Lambda$1492/0x0000000101102440.get(Unknown Source)
java.base@11.0.15/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11.0.15/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
java.base@11.0.15/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
java.base@11.0.15/java.security.AccessController.doPrivileged(Native Method)
java.base@11.0.15/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
java.base@11.0.15/java.lang.Thread.run(Thread.java:829)

This stack trace tells us the cause of the hang is inside the JDK detection routine JdkUtil.checkForJdk (so at least the "detecting JDK" window isn't lying). The Java process is using Windows Native file access method OpenNtQueryDirectoryInformation0. The thread does move between CreateFile0(Native Method) and WindowsNativeDispatcher.OpenNtQueryDirectoryInformation0(Native Method). That tells us this process is moving forward, but just incredibly slowly, so this isn't a (dead or live) lock.

One suspect immediately came to mind is WSL2 (although there is nothing mentioning WSL inside this stacktrace). Recall WSL2 is an underlying Hyper-V virtual machine with a Linux native filesystem and a Windows filesystem translation layer. This meant improved IO speed inside Linux but decreased IO speed from Windows accessing Linux files. Here is the WSL2 documentation from Microsoft describing the issue: "File performance across the Windows and Linux operating systems is faster in WSL 1 than WSL 2, so if you are using Windows applications to access Linux files, you will currently achieve faster performance with WSL 1".

Following my instinct, I opened process explorer in Windows to take a peek at the file handles opened by IntelliJ's process, but that did not show any suspicious file paths that are linked to WSL. But I was not ready to give up my suspicion on IntelliJ WSL integration.

The next hint came from IntelliJ logs in %LOCALAPPDATA%\JetBrains\IdeaIC2022.2\log\. There aren't any useful logs directly related to "detecting JDK". The last thing IntelliJ printed was the following:

2023-xx-xx xx:xx:xx,xxx [  xxxxx]   INFO - #c.i.e.w.WslDistributionManager - Fetched WSL distributions: [(Debian, version=2)] ("C:\Windows\system32\wsl.exe --list --verbose" done in 94 ms)

So immediately before detecting JDK, IntelliJ pulled a list of WSL installations. This is certainly suspicious. The theory I rested on is that the WSL routine found the list of WSL installations, and handed the paths to the JDK detection routine to check for JDK installations. By using Windows file system access methods to check inside a WSL2 installation, translation is happening at an extremely slow pace and "hangs" the JDK detection routine. And because of poor UX designs, the user cannot stop or turn off this process, leading to a major UX issue.

Is Windows Defender involved? In a few comments and support articles (such as here and here), JetBrains employees recommended adding a list of exclusions. IntelliJ even went as far as implementing a feature to prompt the user to "add Windows Defender exclusions for improved IDE performance".

For this, I'll go over a brief overview of antivirus in general (I'm barely an expert myself). AntiVirus, including Windows Defender, has a list of binary patterns of known malicious programs inside its definition file. It checks each file not previously known to it by examining its content against the list of binary patterns using some kind of string matching algorithm. For compressed files (which jars - java archives - are), it will extract the files using the decompression algorithm then do the string matching in memory. This check is done periodically (scheduled scans) and just-in-time when the program is being started (when the user freshly downloaded the files). Decompression and string comparison is known to take time to execute. And to prevent malicious programs from running, the antivirus stalls the file access until it can finish checking the file. The slow scanning algorithms and intentionally stalling file access is what could cause a slow down. But after the antivirus has checked a file once and the file isn't modified, it would save a hash of the file into its "scanned" cache to avoid checking it again, like how incremental compilation is done if a source file is unmodified. So this would only affect IntelliJ on its first launch.

However, this particular problem is not caused by Windows Defender (at best, Defender only exacerbates what is already a severe issue). For one, if the user downloads IntelliJ 2020.3, which does not contain WSL support, the JDK detection process is immediate, even with Defender on. Another proof is that some users including myself are hit with this issue even after disabling Defender. So disabling Windows Defender should not be treated as a silver bullet in all IntelliJ slowness issues.

Workaround - Turning off WSL2 Integration in IntelliJ

Although the above theory is unproven until this point (mainly because I've never worked on IntelliJ's codebase and am not familiar with JDK detection), I'm eager to use it as a basis for experiments to workaround the issue.

The following workaround turns off WSL2 support in IntelliJ by modifying the IntelliJ source code. This workaround is useful for people who are using IntelliJ with Windows JDKs and not interested in Linux JDKs inside WSL2. (For people who wants to use Linux JDKs inside WSL2, do not use this workaround. Run your IntelliJ inside WSL2, and access your IDE through the web frontend, until a proper fix is in place.)

Since IntelliJ community source code is available, I found the class responsible in the log: com.intellij.execution.wsl.WslDistributionManager (Github). The compiled file resides in <intellij-dir>\lib\app.jar for versions 2022.2 and under, and <intellij-dir>\lib\util-8.jar for versions 2023.2 and higher (I didn't bother looking at versions between 2022.3 and 2023.1, it would be one or the other). In theory, we can modify the source code to always return "WSL not supported" and "no WSL installation is found" and recompile the program.

The interested methods are WslDistributionManager.getInstalledDistributions() and WSLUtil.isSystemCompatible(). Make getInstalledDistributions return an empty list and isSystemCompatible to always return false would do the trick. WslDistributionManager.getInstance() is also necessary because it is referenced by other classes. So here are the code you need to write:

public class WslDistributionManager {
    public static WslDistributionManager getInstance() {
        return new WslDistributionManager();
    }

    public List<WSLDistribution> getInstalledDistributions() {
        return List.of();
    }
}

public class WSLUtil {
    public static boolean isSystemCompatible() {
        return false;
    }
}

Ideally, we clone the IntelliJ GitHub repo, replace the above files, then do a recompilation of the whole program. But since I'm short on time, we'll just inject the compiled code for the above classes directly into the java archives (this would cause breakage if you didn't implement all the methods referenced, so I cannot promise it won't break in some codepath). To compile, call javac directly on the .java source files, or use IntelliJ 2020.3 to create a new project and compile the files. The output should be .class files.

Extract app.jar, replace the files WslDistributionManager.class and WSLUtil.class in com\intellij\execution\wsl\. Then compress the extracted directory as a ZIP. Rename it to app.jar, and place it back in IntelliJ's installation directory (keep the original copy by renaming its extension to .bak). Check to make sure the file permissions are the same as the old file permissions.

Then start IntelliJ again, and you should be able to proceed as normal, albeit now IntelliJ won't see any of your WSL2 installations. This means the theory is proven - the slowness is indeed caused by the JDK detection routine interacting with WSL2.

How JetBrains Engineers can Fix the Problem

The problem seems to be extreme slowness when accessing the WSL2 file system through Windows file system APIs using the compatibility layer built by Microsoft. Detecting JDK is one such user of this API.

According to engineer Serge Baranov in this comment, JetBrains is aware of the technical differences between having a server running in WSL2 (also the approach VSCode takes) compared to accessing WSL2 through the Windows filesystem API. The server approach bypasses the Windows filesystem API and accesses the files inside Linux directly. The processed data is then passed from Linux to Windows over a network connection. This would be the ultimate solution for people who are interested in development inside WSL2, obsoleting the Windows API codepath.

But if JetBrains engineers are not ready to abandon the Windows version of IntelliJ on Windows with WSL2 integration, then an alternative can be taken to improve JDK detection performance. Rather than using Windows APIs from Java to detect JDKs, a Linux shell script can be written and run inside Linux. The output of the script can be written as a file into IntelliJ's local app data Windows directory. By doing JDK detection inside Linux, native IO performance can be obtained, and this issue should no longer manifest itself.

Wrap

IntelliJ versions 2021.1 - 2023.2 has WSL2 support, but this will negatively impact certain system configurations due to poor WSL2 <-> Windows cross filesystem performance. There are a few UX improvements to be made, as well as a deeper fix for the root issue. Meanwhile, there is a hack end users can execute to disable WSL2 support in IntelliJ to workaround the issue.

Fixing iOS Simulator Black Screen on NVidia Kepler Macs with OCLP

Yang Fang — Tue, 10 Oct 2023 16:52:43 +0000

Background

I've previously shared how to develop and debug iOS 16 apps on macOS catalina. While that workaround allowed compiling code with a newer toolchain than officially supported, Catalina proved to be too old of a system to effectively support modern iOS development. A huge limitation is its limit to Xcode 12.4.

I've finally made the decision to use Opencore Legacy Patcher to install newer macOS on my iMac late 2013. Although OCLP officially supports macOS 14, there are still quite a few outstanding bugs. That calls for being more conservative when considering upgrading macOS on legacy Macs. So I picked MacOS 12 as the base for my iMac late 2013.

The way OCLP works is by extracting kexts (or in more common terminology - drivers) from older macOS releases and install them in the new macOS version. Because this combination (older kexts from old OS on a new OS) is not supported, you can run into bugs that does not exist on supported configurations.

Problem

My particular iMac has an NVidia Kepler graphics card. NVidia had a row with Apple causing Apple to switch to AMD in later Macs (despite NVidia being superior in energy efficiency for much of the 2014-2020 period).

The conflict between the two companies shows in the quality of the NVidia Metal driver. The NVidia driver has trouble running the iOS 16.2 simulator that comes with XCode 14.2 (the last XCode version supported by macOS 12). When the simulator is started, the screen remains black, although the log shows the simulator processes are running fine.

This problem is only relevant to people with NVidia Kepler graphics cards. NVidia GPUs before Kepler do not support metal. Later Macs use AMD. And Macs of the same generation using Intel Haswell processors integrated graphics is also free of this problem. This is one peculiar bug that affect a very small subset of people.

Through experimentation, I determined the last release of iOS simulator that works with Kepler driver is the iOS 15.2 simulator bundled with XCode 13.2.1. If you do not mind the extra disk space, and do not mind testing with iOS 15 rather than iOS 16, then simply download the iOS 15.2 runtime in Xcode and everything should work.

Here is the workaround I discovered after some experimentation to get 16.2 simulator working. (I've tested it on macOS 12 and XCode 14.2. It should also work on macOS 13 with XCode 14.x. I have not tested this on XCode 15 / iOS 17 Simulator, or on macOS 14.)

Obtaining iOS 15.2 Simulator

To start, you need to download the iOS 15.2 simulator. There are two ways to do it:

Download iOS 15.2 Simulator in XCode

In XCode -> Preferences -> Platforms -> ➕ -> iOS -> iOS 15.2 Simulator.

Then, show package content of the simulator runtime. It is in /Library/Developer/CoreSimulator/Profiles/Runtimes if you installed Simulator through XCode.

Obtain iOS 15.2 Simulator from Apple Developer Website

If you do not want to download the runtime inside XCode, which is much smaller, you can also download XCode 13.2.1 from the Apple Developer website.

After you extract the package, show package content and find the runtime in Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes.

After Obtaining iOS 15.2 Simulator

Also open up the content of XCode 14.2, navigate to Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes to find the iOS 16.2 runtime.

The magic files you need from iOS 15.2 runtime is Resources/RuntimeRoot/System/Library/PrivateFrameworks/MTLSimDriver.framework. Copy and replace the folder in the same path from the iOS 15.2 runtime to the iOS 16.2 runtime.

After this, quit iOS 16.2 Simulator if you have it open, and start it up again. This time, the screen should no longer be black, because by replacing this private library, iOS 16.2 Simulator can now draw using the older API.

Working on iOS 15+ Code on MacOS Catalina

Yang Fang — Tue, 29 Aug 2023 05:43:16 +0000

I try very hard to keep macOS Catalina, which my iMac late 2013 was stuck on, going for iOS development with Flutter. I've previously shared various workarounds that allowed compiling Flutter app successfully and deploying to a real iOS device using updated device support files. Today, I am back with yet another workaround that is not previously discovered.

With the previous method, we had to delete references to any new API introduced in iOS 15 or later so we can use the iOS 14 toolchain to compile it. Otherwise, the toolchain will indicated an error. That may not work if you need to depend on closed source components where you cannot change the source code.

This new workaround goes a little further. It will actually compile iOS 15+ code using iOS 15+ toolchains and SDKs. This method does not require us to modify any (open source) library code we reference, and also helps us compile against closed source libraries where we do not have access to the source code.

Note: the best way to develop for iOS is to use the latest Xcode on the latest macOS. And on an unsupported Mac it means using OCLP (OpenCore Legacy Patcher). That will ensure the best development experience without much tinkering. This guide is for people who for whatever reason do not want to use OCLP or want to remain on macOS Catalina.

But I must warn you that this workaround comes with significant drawbacks. Here are the pros and cons up front before you invest too much time:

Pros:

Compile iOS 15+ code using iOS 15+ SDK on Catalina w/ Xcode 12.4
Debug code compiled with the iOS 15+ toolchain on the iOS 14.4 simulator that comes with Xcode 12.4

Cons:

Cannot deploy code compiled with iOS 15+ SDK on a real device
Cannot test code compiled with iOS 15+ SDK on iOS 15+ simulator (because the iOS 15+ simulators will not run on Catalina)
Cannot submit compiled code to App Store (You will need the latest Xcode to do this!)
Xcode sometimes bugs out and uses the wrong SDK to compile, resulting in annoying errors

You've been warned, now let's dive in.

Download Xcode

In order to get newer iOS SDKs and compilers, we need newer copies of Xcode. This can be downloaded from Apple's developer website as long as you have an Apple ID. No paid membership is required.

Head to the download section of the developer website, then check the Xcode downloads. I recommend only using stable versions of Xcode. For this example, I will use Xcode 14.2 (which comes with iOS 16.2 SDK).

Wikipedia has a helpful page on Xcode compatibility with various macOS versions and the SDKs they come with.

After you have the xip file, double click it to extract it as a macOS program.

Copy SDK

Xcode 14.2 runs natively on macOS 12.5, so it will not run under macOS Catalina. We will copy files from inside its content, similar to what we did with Device Support files in a previous method.

Right click on the Xcode program, select "Show Package Contents", then go into Contents/Developer/Platform. Here you will see the various SDKs Xcode 14.2 comes with. We will need both iPhoneOS.platform and iPhoneSimulator.platform for iOS. You may also collect SDKs for other platforms such as tvOS, but I have not tested them.

Go into iPhoneOS.platform/Developer/SDKs, rename iPhoneOS.sdk to iPhoneOS16.2.sdk (you can delete the symlink with the same name, we won't need it), then copy to your Xcode 12.4 installation (the default would be your Application folder: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform). After copying, your Xcode 12.4 folder should have iPhoneOS.sdk, iPhoneOS14.4.sdk (symlink), iPhone16.2.sdk (the one we copied).

Do the same for iPhoneSimulator.platform.

After this step, your Xcode 12.4 will see new SDKs, although you cannot compile code with it yet. That's because the SDK will use new language features introduced in newer Swift versions, and the default toolchain (compiler, linker, etc) will not be able to understand the new Swift features.

Copy Toolchain

We will now copy the new toolchain. Toolchain is the collection of utility to compile and package your code into a testable, deployable, or uploadable form. This includes the Swift tools (based on LLVM).

Normally, you can download additional toolchains from swift.org. These toolchains are the open source versions of Swift the Xcode toolchains are based on. If you install any open source toolchains, they will be installed at /Library/Developer/Toolchains/. Xcode is designed to work with the open source toolchains, which we are taking advantage here. Although this feature also comes with the limitation that you cannot deploy any compiled binaries onto a real device (and thus our workaround will be subject to the same limitation).

From the Xcode 14.2 base, go into Contents/Developer/Toolchains. You will see XcodeDefault.xctoolchain. Copy this to /Library/Developer/Toolchains/ and rename it to something more descriptive and to your liking. For example, I use Xcode-14.2.xctoolchain.

You cannot use this toolchain without performing one further step, which is to fool Xcode into thinking it is one of the open source toolchains from swift.org.

Right click on the xctoolchain file and select "Show Package Contents". The Xcode toolchain comes with a file ToolchainInfo.plist, which is not useful for our purpose. We will delete it. We will create a file called Info.plist. I provided the content below. (Another way to obtain this file is to download the corresponding open source toolchain and copy over this file).

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Aliases</key>
    <array>
        <string>swift</string>
    </array>
    <key>CFBundleIdentifier</key>
    <string>org.swift.573202201171a</string>
    <key>CompatibilityVersion</key>
    <integer>2</integer>
    <key>CompatibilityVersionDisplayString</key>
    <string>Xcode 8.0</string>
    <key>CreatedDate</key>
    <date>2023-01-15T23:17:44Z</date>
    <key>DisplayName</key>
    <string>Swift 5.7.3 Release 2022-01-17 (a)</string>
    <key>OverrideBuildSettings</key>
    <dict>
        <key>ENABLE_BITCODE</key>
        <string>NO</string>
        <key>SWIFT_DEVELOPMENT_TOOLCHAIN</key>
        <string>YES</string>
        <key>SWIFT_DISABLE_REQUIRED_ARCLITE</key>
        <string>YES</string>
        <key>SWIFT_LINK_OBJC_RUNTIME</key>
        <string>YES</string>
        <key>SWIFT_USE_DEVELOPMENT_TOOLCHAIN_RUNTIME</key>
        <string>YES</string>
    </dict>
    <key>ReportProblemURL</key>
    <string>https://bugs.swift.org/</string>
    <key>ShortDisplayName</key>
    <string>Swift 5.7.3 Release</string>
    <key>Version</key>
    <string>5.7.3.20220117101</string>
</dict>
</plist>

The exact names inside this file does not matter too much. You can for example change the ShortDisplayName to Xcode 14.2 to better identify it in Xcode.

After all is done, Xcode will be able to see the toolchain as if it is one of the open source toolchains, and can use it to compile code using the corresponding SDK.

Doing Compilation in Xcode

Open Xcode 12.4, at the top menu bar, under "Xcode -> Toolchains", you should see a new toolchain listed. Select it.

Open your project if you haven't already, open the project file and go into "Build Settings", under "Base SDK", you should be able to choose between SDKs (such as iOS, macOS, tvOS, etc) and their versions. Xcode by default hides the highest version (the iOS 16.2 version we copied) because it assumes the base version iOS would be the highest. Since we didn't change the symlink in the SDK folder, our default iOS SDK is the same as iOS 14.4. Regardless, select "Other..." and type in iphoneos16.2. This will instruct Xcode to compile the project with 16.2.

The next step is just to compile the project for running on the simulator. You may need to clean the build folder and Xcode cache or even restart Xcode or the computer, since Xcode may be confused about files produced by an old toolchain. I have a lot of problems with switching between the newer toolchains + SDKs and the default toolchain + SDK. Sometimes Xcode will be using the default toolchain to compile with newer SDKs despite the project file saying otherwise.

This is the end of the tutorial.

Afterword

This is probably the closest you can get to be compiling iOS 15+ code on macOS 10.15 Catalina. It works if your primary development flow relies heavily on the simulator and you don't need to test iOS 15 integrations.

Apple has also been making a lot of changes to the new Xcode 15 (iOS 17 SDK) architecture, including changing the device stack for iOS (no more device support files). The new swift toolchain (5.9 beta) also no longer runs on Catalina. This means the end of the road for Catalina is coming in a few months even if you managed to use this workaround.

You're now strongly advised to seek other alternatives, such as a new mac, OCLP, or a macOS VM. It will give you less headache and more time for focusing on development. Either way, I hope you enjoyed this tutorial.

Common Errors Compiling New Flutter Apps on MacOS Catalina

Yang Fang — Mon, 03 Apr 2023 18:09:50 +0000

If you followed my previous two guides, you can now compile Flutter apps on MacOS Catalina (despite Flutter restrictions that require you to upgrade your MacOS version). However, you may still run into a few compilation errors.

pod not found

Sometimes you will encounter a build error in XCode complaining about pods. You have to go to the ios folder in your project and run pod install in the terminal.

Error when reading 'lib/main.dart'

If your flutter entry point is not main.dart (such as using project templates that uses different flavor names), you will need to point to the entry point file in "Build Settings". Under "Build Settings" there is "FLUTTER_TARGET" which should point to your entry point file such as lib/main_development.dart.

Property 'xxx' not found on object of type 'XXX *'

Since XCode 12.4 comes with iOS 14.4 SDK, code written for the newer SDK assuming availability of iOS 15 APIs will fail to compile. The errors look like the following:

Semantic Issue (Xcode): Property 'interruptionLevel' not found on object of type 'UNMutableNotificationContent *'
{Source File Path}:{Line Number}

If you look into the offending source file, you'll likely see usage of library API introduced in iOS 15. The call should already be wrapped with if (@available(iOS 15, tvOS 15, *)), guarding it against runtime errors should the application run on a lower iOS version, but the compilation error still happens if you compile the code with iOS 14 SDK.

Despite my efforts of copying SDK from a new XCode build, I cannot get XCode 12.4 to recognize newer iOS SDK. (If anyone know how, please let me know in the comments)

The workaround is to simply comment out the whole block inside if (@available(iOS 15...)) block. This means regardless of whether iOS 15 is available, we will not execute code for iOS 15. This should be fine for testing purposes. But I reiterate my recommendation to use a virtual machine to build the app properly for your production builds.

Deploying Flutter to iOS 15 on MacOS Catalina

Yang Fang — Fri, 03 Feb 2023 06:03:27 +0000

Continuing from the last post, where I discussed how to work around Flutter's XCode version requirement, in this post, I will discuss how to deploy a test iOS App on a real iOS device (a.k.a. iPhone).

Similar to Flutter, XCode also has compatibility requirements between XCode versions and deployable iOS versions. With Catalina, the latest supported XCode version is 12.4, and the latest supported iOS version is iOS 14.4. This poses a problem, because almost all modern iPhones currently run a later version of iOS. Even an "ancient" iPhone 6s from 2015 runs iOS 15.7 today. Since Apple does not sign old iOS versions, users are unable to downgrade to previous iOS versions (outside of jailbreaking). Thus, current iPhone devices can't be used for testing.

Over the past few years, various people on the Internet have figured out workarounds to specific issues arising from these restrictions. I have compiled a step by step guide below and will reference these posts when I go over what changes needed to be made to a default XCode 12.4 installation on macOS Catalina with Flutter 3.3.10 in order to deploy to an iPhone 6S running iOS 15.7.

As a prerequisite, you'll want to make sure Flutter 3.3.10 builds your project and deploys to the iOS simulator (simulating iOS 14.4). This should be the case if you followed my previous post.

Now we will connect the iPhone 6S to the Mac running macOS Catalina. You may be prompted to install software components to sync this iOS version (this doesn't have anything to do with XCode, just normal iTune/Finder sync), which I recommend installing. After that, you should see your iPhone show up in Finder.

Download Device Support Files

[Reference]

Since we want to deploy to iOS 15.7, we will need the Device Support files for this iOS version. Head to https://github.com/iGhibli/iOS-DeviceSupport and download DeviceSupport/15.7(FromXcode_14.1_Release_Candidate_xip).zip, unzip this file in Finder, then move the folder 15.7 to /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport.

If you're deploying to iOS 16, then download Device Support for iOS 16.1, and rename the folder to the corresponding iOS version on your device.

Update Signing Config in XCode

In order to deploy to a real device, the app needs to be signed using an Apple ID (a.k.a. iCloud login), which is free. App signing is normally done by XCode.

We will then open the Flutter iOS project in XCode. Remember this is the ios/Runner.xcworkspace file and not the ios/Runner.xcodeproj file.

In the project navigator, click on the Runner project. This opens Runner.xcodeproj properties. On the tab bar (where it shows "Info", "Build Settings", etc), make sure that the Runner target is selected (not Runner project). Go to "Signing & Capabilities". Here you will see that Team is set to none. Select the dropdown and add an account. A preference window will open. Sign in to your Apple ID in the preference window.

After that, close out of the preference window and select the signed in account in Team. If you see a message that says no matching device is available, click the device name from the XCode title bar "Runner > (device name)" drop down menu (this is located at the top of the window next to the ▷ and □ buttons). Then, select iPhone 6s in the dropdown. This should register the iPhone with XCode and get a provisioning profile for 7 days.

Update More Signing Config in XCode

[Reference]

We are not quite done as there is one more hurdle. Apple changed the signature format, and requires an additional commandline change. Otherwise the deployment will fail with a signature error.

In Runner.xcodeproj properties, make sure that the Runner target is selected. Go to "Build Settings" tab. Select the "All" sub-tab. Search for "Other Code Signing Flags". Add --generate-entitlement-der in the field.

Now you should finally be able to deploy to iPhone 6S on XCode 12.4 on macOS Catalina.

Workaround for New Flutter Version on MacOS Catalina

Yang Fang — Mon, 09 Jan 2023 18:44:28 +0000

The Flutter SDK is a development framework that allows developers to write code once and run on multiple platforms such as mobile, web, and desktop.

In order to do iOS development, Flutter must be run on MacOS with an XCode installation. Since Flutter 3.0, XCode 13 is required. XCode 13 is not supported on MacOS Catalina (12.4 is the last XCode supported on MacOS Catalina). This means you can no longer do Flutter 3.0 development on MacOS Catalina.

Not everyone can afford to upgrade to Apple's pricy hardware. (Although the newly released M2 Mac mini makes it the cheapest Apple hardware to date, there are still a lot of folks in other parts of the world where household income is not high) For the developers with an old (and unsupported) Mac, this is frustrating.

Host Mac Version	Highest XCode	Target MacOS Version	Target iOS Version	Flutter Version	Dart SDK Version
10.15	12.4	11.1	14.4	2.10	2.16
11	13.2.1	12.1	15.2	3.0	2.17

I discovered a workaround that allows Flutter 3 to continue building on MacOS Catalina, but keep in mind this is not supported by the Flutter team and may cause unexpected issues.

The XCode version check is done in packages/flutter_tools/lib/src/macos/xcode.dart inside the Flutter SDK root folder. To reverse the check, we need to do something opposite of what https://github.com/flutter/flutter/pull/97746 did.

For Flutter 3.0.5:

--- a/packages/flutter_tools/lib/src/macos/xcode.dart
+++ b/packages/flutter_tools/lib/src/macos/xcode.dart
@@ -18,11 +18,11 @@ import '../base/version.dart';
 import '../build_info.dart';
 import '../ios/xcodeproj.dart';

-Version get xcodeRequiredVersion => Version(13, null, null);
+Version get xcodeRequiredVersion => Version(12, 3, null, text: '12.3');

 /// Diverging this number from the minimum required version will provide a doctor
 /// warning, not error, that users should upgrade Xcode.
-Version get xcodeRecommendedVersion => xcodeRequiredVersion;
+Version get xcodeRecommendedVersion => Version(13, null, null, text: '13');

 /// SDK name passed to `xcrun --sdk`. Corresponds to undocumented Xcode
 /// SUPPORTED_PLATFORMS values.

For Flutter 3.3.10 / 3.7.9 / 3.10.6

--- a/packages/flutter_tools/lib/src/macos/xcode.dart
+++ b/packages/flutter_tools/lib/src/macos/xcode.dart
@@ -18,7 +18,7 @@ import '../base/version.dart';
 import '../build_info.dart';
 import '../ios/xcodeproj.dart';

-Version get xcodeRequiredVersion => Version(13, null, null);
+Version get xcodeRequiredVersion => Version(12, 3, null, text: '12.3');

 /// Diverging this number from the minimum required version will provide a doctor
 /// warning, not error, that users should upgrade Xcode.

For Flutter 3.13.0

--- a/packages/flutter_tools/lib/src/macos/xcode.dart
+++ b/packages/flutter_tools/lib/src/macos/xcode.dart
@@ -20,7 +20,7 @@ import '../base/version.dart';
 import '../build_info.dart';
 import '../ios/xcodeproj.dart';

-Version get xcodeRequiredVersion => Version(14, null, null);
+Version get xcodeRequiredVersion => Version(12, 3, null, text: '12.3');

 /// Diverging this number from the minimum required version will provide a doctor
 /// warning, not error, that users should upgrade Xcode.

Dart compiles the source on the fly when required, but if cache is available, it will use the compiled cache which will not contain our version check hack. You will need to delete the following file after making the above change:

rm bin/cache/flutter_tools.snapshot

After that, you can run your flutter normally. The first run will re-compile the flutter tools package with our version check hack.

This workaround merely disables the version check. As I can confirm with my experience with Flutter 3.0.5, 3.3.10, and 3.7.9, this change does not cause any major issue with building Flutter Apps for iOS for both the iOS emulator and for a real device. However, do note that if features relying on the newer XCode version are added, your build will break.

I would only recommend using this workaround for development. If you need to produce production builds, I strongly recommend using a virtual machine with the latest MacOS version (which is painful to setup and use, but will nonetheless be able to run the latest XCode and Flutter "natively").

For now though, you can use continue to use your outdated Mac for a little longer for development and testing purposes.

PS: Yet another workaround is to upgrade your unsupported Mac to Big Sur/Monterey using OCLP. But at least for my iMac with an NVidia GPU, that configuration requires a little extra effort to inject the NVidia drivers and is less stable. So a few line changes in the SDK is a bit simpler for me.