DEV Community

Sifiso Fakude
Sifiso Fakude

Posted on

Building a Plug-and-Play JVM Compiler for Android and Desktop with Bytesmith

#android #java #kotlin #software

What if adding Kotlin and Java compilation to your application didn't mean building an entire compilation pipeline yourself?

What if you could add Bytesmith, configure the filesystem once, provide your source files and output destination, and simply compile?

That's the idea behind Bytesmith.

Bytesmith is a Kotlin and Java compiler toolkit designed for JVM and Android applications. It provides a unified API for Kotlin, Java, and mixed-language compilation, while also supporting filesystem abstraction, custom classpaths, boot classpaths, compiler plugins, packaging, and diagnostics.

Configure the environment, provide the source, specify the output, and compile.

The problem

Compiler tooling can become surprisingly difficult when it is tightly coupled to the environment in which it was originally designed to run.

You might need to deal with:

  • Kotlin compiler versions
  • Kotlin standard libraries
  • Java compilation
  • Bootclasspath configuration
  • Dependency classpaths
  • Source discovery
  • Output handling
  • Android storage
  • Storage Access Framework URIs
  • Packaging
  • Compiler diagnostics

And then there is the question of where those files actually live.

On a desktop JVM, you might have traditional filesystem paths:

/home/user/project/src/Main.kt
Enter fullscreen mode Exit fullscreen mode

On Android, you might be working with application storage or files selected through the Storage Access Framework:

content://...
Enter fullscreen mode Exit fullscreen mode

If your compiler API directly depends on java.io.File, your compilation code becomes coupled to one filesystem model.

Bytesmith takes a different approach.

Adding Bytesmith

The goal is to make compilation something you can plug into an application.

With Gradle:

implementation("io.github.sifisofakude.bytesmith:bytesmith-common:1.0.0")
Enter fullscreen mode Exit fullscreen mode

After adding Bytesmith, configure the filesystem your application wants to use.

For a JVM application:

FileSystems.current = JvmFileSystem()
Enter fullscreen mode Exit fullscreen mode

For Android:

FileSystems.current = AndroidSafFileSystem(context)
Enter fullscreen mode Exit fullscreen mode

Once the filesystem is configured, the rest of the compilation layer can operate through the filesystem abstraction.

The basic compilation example

Bytesmith provides a high-level compilation entry point.

For example:

val result = Main().compile(
    JvmFileSystem(),
    listOf(
        "-d", "build/classes",
        "src/main/java",
        "src/main/kotlin"
    )
)

println(result)
Enter fullscreen mode Exit fullscreen mode

The important part is that you provide the filesystem and compilation arguments.

Bytesmith handles the compiler orchestration.

You don't need to build an entire compilation pipeline yourself.

Compiling Kotlin

Bytesmith also exposes the Kotlin compiler directly when you need more control.

For example:

val result = KotlinCompiler(JvmFileSystem()).compile(
    Options(
        outputDir = "build/classes",
        kotlinSources = kotlinSources,
        javaSources = emptyList(),
        classpath = emptyList(),
        bootClasspath = emptyList(),
        warningsAsErrors = false
    )
)
Enter fullscreen mode Exit fullscreen mode

This gives your application control over the compilation environment while still allowing Bytesmith to handle the underlying compiler integration.

Compiling Java

Bytesmith isn't just a Kotlin compiler wrapper.

Java compilation is supported through ECJ.

For example:

val result = JavaCompiler(JvmFileSystem()).compile(
    Options(
        outputDir = "build/classes",
        javaSources = javaSources,
        classpath = emptyList(),
        bootClasspath = emptyList(),
        warningsAsErrors = false
    )
)
Enter fullscreen mode Exit fullscreen mode

This means an application can use Bytesmith for Java compilation without implementing its own ECJ integration.

Mixed Kotlin and Java projects

Bytesmith can also compile projects containing both Kotlin and Java source files.

For example:

src/
└── main/
    ├── kotlin/
    │   └── Example.kt
    └── java/
        └── JavaExample.java
Enter fullscreen mode Exit fullscreen mode

You can provide both types of source files to the compilation process:

val sources = listOf(
    "src/main/kotlin/Example.kt",
    "src/main/java/JavaExample.java"
)
Enter fullscreen mode Exit fullscreen mode

The Kotlin compiler and Java compiler are orchestrated as part of the compilation process.

This is useful for applications that generate source code, developer tools, code editors, custom build systems, or other environments where compilation itself is an application feature.

The filesystem abstraction

This is one of the most important parts of Bytesmith.

A compiler shouldn't necessarily care where a source file physically comes from.

A desktop application might use:

/home/user/project/Main.kt
Enter fullscreen mode Exit fullscreen mode

An Android application might use application storage.

Or the source might come from a user-selected SAF document:

content://...
Enter fullscreen mode Exit fullscreen mode

These are fundamentally different storage mechanisms.

But the compilation operation doesn't need to be.

The application chooses the filesystem implementation:

FileSystems.current = JvmFileSystem()
Enter fullscreen mode Exit fullscreen mode

or:

FileSystems.current = AndroidSafFileSystem(context)
Enter fullscreen mode Exit fullscreen mode

Bytesmith then works through the filesystem abstraction.

The architecture looks roughly like this:

             Application
                  |
                  v
          FileSystems.current
                  |
        +---------+---------+
        |                   |
        v                   v
 JvmFileSystem       AndroidSafFileSystem
        |                   |
        +---------+---------+
                  |
                  v
              Bytesmith
                  |
          +-------+-------+
          |               |
          v               v
     Kotlin compiler     ECJ
          |               |
          +-------+-------+
                  |
                  v
              Output
Enter fullscreen mode Exit fullscreen mode

The compiler doesn't have to know whether the underlying storage is a JVM filesystem or SAF.

Android and SAF

This becomes particularly useful on Android.

The Storage Access Framework doesn't behave like a traditional filesystem.

Instead of receiving a normal filesystem path, your application may receive a URI:

content://...
Enter fullscreen mode Exit fullscreen mode

You may then need to work with:

  • ContentResolver
  • Document providers
  • Document tree URIs
  • Persisted permissions
  • DocumentFile
  • DocumentsContract

Bytesmith's filesystem abstraction allows those details to stay inside the filesystem implementation.

The compilation code doesn't need to become filled with Android-specific storage logic.

You configure the filesystem once.

Then Bytesmith can work through it.

Scoped Storage, SAF, or JVM paths?

From the perspective of the compilation API, it shouldn't fundamentally matter whether your application is using:

JVM filesystem
    |
    +-- /home/user/project/Main.kt
Enter fullscreen mode Exit fullscreen mode

or:

Android application storage
    |
    +-- /data/user/0/...
Enter fullscreen mode Exit fullscreen mode

or:

Storage Access Framework
    |
    +-- content://...
Enter fullscreen mode Exit fullscreen mode

Those are different storage mechanisms.

They don't have to result in completely different compiler implementations.

The filesystem implementation handles the difference.

That's one of the reasons I built the filesystem abstraction separately from the compilation logic.

Android usage

An Android application can configure the appropriate filesystem implementation:

FileSystems.current = AndroidSafFileSystem(context)
Enter fullscreen mode Exit fullscreen mode

After that, the same Bytesmith compilation layer can operate against the configured filesystem.

If SAF resources need to be materialized for a particular compilation operation, the filesystem implementation can handle that as well.

For example:

val local = fs.materialize(uri, "Sources")
Enter fullscreen mode Exit fullscreen mode

and later:

fs.clearMaterialized("Sources")
Enter fullscreen mode Exit fullscreen mode

The important thing is that the compilation layer doesn't need to implement its own SAF handling.

Classpaths

Real compilation usually requires dependencies.

Bytesmith allows you to provide a custom classpath.

For example:

val options = Options(
    outputDir = "build/classes",
    kotlinSources = kotlinSources,
    javaSources = javaSources,
    classpath = listOf(
        "libs/library.jar",
        "libs/another-library.jar"
    ),
    bootClasspath = emptyList(),
    warningsAsErrors = false
)
Enter fullscreen mode Exit fullscreen mode

The application decides what libraries should be visible to the compiler.

Bytesmith handles passing those dependencies into the appropriate compiler.

Kotlin standard library versions

This separation is also useful when working with Kotlin's standard library.

The Kotlin compiler and Kotlin standard library are related, but they aren't exactly the same thing.

If your generated source needs APIs from a newer Kotlin standard library, you can provide the appropriate:

kotlin-stdlib.jar
Enter fullscreen mode Exit fullscreen mode

through the compilation classpath.

For example:

val classpath = listOf(
    "libs/kotlin-stdlib.jar",
    "libs/my-library.jar"
)
Enter fullscreen mode Exit fullscreen mode

This means an update to the standard library doesn't automatically mean Bytesmith itself needs to be updated simply because the library JAR changed.

There is an important limitation.

Providing a newer kotlin-stdlib.jar does not make an older Kotlin compiler understand new Kotlin language features.

If a newer Kotlin release introduces new syntax, compiler functionality, or lowering requirements, the compiler itself needs to support those features.

The goal is not to pretend compiler versions don't matter.

The goal is to avoid coupling things that can evolve independently.

Bootclasspath

Compilation environments sometimes require a bootclasspath.

Bytesmith can detect the appropriate bootclasspath where possible.

If you need explicit control, you can provide one yourself.

This gives you two options.

Simple case

Let Bytesmith determine the environment.

Advanced case

The application supplies the bootclasspath.

You get convenience without losing control.

Compiler plugins

Bytesmith also supports Kotlin compiler plugins.

You can provide plugin JARs and plugin options as part of the compilation environment.

For example:

val options = Options(
    pluginClasspath = listOf(
        "plugins/my-plugin.jar"
    ),
    pluginOptions = listOf(
        "plugin:my.plugin:key=value"
    )
)
Enter fullscreen mode Exit fullscreen mode

This makes Bytesmith useful for more than straightforward source compilation.

An application can construct a compilation environment specifically for the code it needs to compile.

Packaging the output

Compilation doesn't necessarily have to end with a directory containing .class files.

Bytesmith can also package compiled output into JAR or ZIP archives.

For example:

build/classes
Enter fullscreen mode Exit fullscreen mode

can be used as a directory output, while an application can also produce:

app.jar
Enter fullscreen mode Exit fullscreen mode

or:

app.zip
Enter fullscreen mode Exit fullscreen mode

This is useful when the compiled result needs to be consumed as an artifact rather than simply left in a classes directory.

Diagnostics

Compilation errors are only useful if your application can actually report them.

Bytesmith exposes compilation results and listener callbacks.

For example:

class Listener : ICompilationListener {

    override fun hasErrors() = false

    override fun onProblem(problem: CompilationProblem) {
        problem.printMessage()
    }

    override fun onClassCompiled(compiledClass: CompiledClass) {
        println(compiledClass.fileName)
    }
}
Enter fullscreen mode Exit fullscreen mode

The compilation result can also provide information such as:

println(result.success)
println(result.errorCount)
println(result.warningCount)
println(result.compiledClassCount)
println(result.elapseTimeMillis)
Enter fullscreen mode Exit fullscreen mode

This becomes particularly useful when Bytesmith is embedded inside a UI.

An Android code editor, for example, could take compiler diagnostics and display them directly to the user rather than dumping compiler output to a terminal.

Command-line usage

Bytesmith isn't limited to an embedded API.

It also provides a command-line interface.

A compilation can look like:

bytesmith   -cp libs/*   -bc jmods   -d build/classes   src/main/java   src/main/kotlin
Enter fullscreen mode Exit fullscreen mode

The CLI exposes options for classpaths, boot classpaths, module paths, source paths, compiler plugins, plugin options, output destinations, and warnings-as-errors.

This means the same compilation capabilities can be used programmatically or from the command line.

Why not just use Gradle?

Gradle is excellent when you're building a conventional project.

But that's not necessarily the problem Bytesmith is trying to solve.

Imagine an application where the application itself needs to compile source code.

For example:

  • An Android IDE
  • A code editor
  • A developer tool
  • A code generator
  • A scripting environment
  • A custom build tool
  • An educational programming environment
  • An application that generates Kotlin or Java code dynamically

In these situations, starting a complete Gradle build just to compile a set of source files can be excessive.

You may simply want:

Source files
     |
     v
  Compile
     |
     v
Classes / JAR
Enter fullscreen mode Exit fullscreen mode

Bytesmith provides that compilation layer.

The architecture

The architecture can be summarized as four responsibilities.

1. Your application

Decides what it wants to compile and where the result should go.

2. The filesystem abstraction

Decides how files are actually accessed.

JvmFileSystem
AndroidSafFileSystem
Enter fullscreen mode Exit fullscreen mode

3. Bytesmith

Orchestrates compilation, classpaths, compiler configuration, diagnostics, and packaging.

4. Language compilers

Kotlin is compiled through the Kotlin compiler.

Java is compiled through ECJ.

This separation is what makes the whole thing composable.

The result

For a simple use case, the workflow is essentially:

Configure filesystem
        |
        v
Provide sources
        |
        v
Compile
        |
        v
Get result
Enter fullscreen mode Exit fullscreen mode

For advanced applications, you can progressively add:

Classpath
Bootclasspath
Compiler plugins
Plugin options
Diagnostics
Packaging
Enter fullscreen mode Exit fullscreen mode

without having to replace the underlying compilation architecture.

That's the balance Bytesmith is aiming for:

Simple when you need simple. Configurable when you need control.

Final thoughts

I built Bytesmith around a fairly simple idea:

Compilation should be a capability an application can plug in, not an entire infrastructure the application has to reinvent.

It can compile Kotlin.

It can compile Java.

It can handle mixed Kotlin/Java projects.

It can work with custom classpaths and boot classpaths.

It supports compiler plugins and packaging.

And, perhaps most importantly for Android, it doesn't assume that every file is a traditional JVM filesystem path.

The application chooses the filesystem implementation.

Bytesmith works through that abstraction.

So whether you're compiling files from a desktop filesystem, Android application storage, or resources exposed through the Storage Access Framework, the compilation layer can remain the same.

Configure the filesystem.

Give Bytesmith the source.

Configure dependencies when necessary.

Choose the output.

Compile.

That's Bytesmith.

Top comments (1)

Collapse
 
vmodal_ai profile image
vmodal_ai •

Great one !