When a Java application needs AI inference, the easiest answer is often to call a remote service. That is practical, but it is not the only direction the Java platform is exploring. A recent Devoxx Belgium session by Java developer advocate Ana-Maria Mihalceanu puts two less familiar paths side by side: use the Foreign Function and Memory API to connect Java to native inference runtimes today, or use Project Babylon's code reflection to express model logic in Java and translate it for other execution engines later.
Those paths solve different problems. FFM is about making the boundary between Java and native code more manageable. Babylon is about making Java code itself available for analysis and transformation, so it can reach programming models that do not execute ordinary JVM bytecode. Treating them as interchangeable would lead to the wrong architecture decision.
The distinction matters because Java's AI story is often reduced to a list of frameworks that call model APIs. That remains useful, especially for orchestration and application integration. But the deeper platform question is where inference runs, what representation the model uses, and how much foreign-runtime glue an application must own. Mihalceanu's session description gives a useful map of those choices, while the OpenJDK project material shows why one path is ready for practical evaluation and the other is still experimental.
The first track is interoperability, not a new Java model format
The Foreign Function and Memory API, or FFM, is a standard Java API for calling native functions and accessing memory outside the JVM. It became final in JDK 22. The FFM route does not mean that Java has suddenly learned every native library's interface, nor that a model file disappears. It means Java has a supported way to link native code and work with native memory without relying on JNI as the only bridge.
In her Devoxx session, Mihalceanu pairs FFM with jextract, a tool that reads C header files and generates Java bindings. The official jextract guide describes the output as Java code that uses FFM internally. That can reduce the hand-written binding layer between an application and a C library. The session description names ONNX Runtime as an example of the kind of AI inference runtime developers might reach through this approach.
The practical mental model is straightforward: Java owns application logic, while a native runtime owns some lower-level inference work. The application still needs to manage the runtime library, model artifacts, native memory and platform compatibility. FFM improves the Java-to-native boundary; it does not eliminate the boundary.
That is a meaningful improvement over brittle integration code, but it is not a magic performance switch. A Java service can pay costs in data conversion, memory ownership, model loading and deployment packaging. Moving one call across a native boundary does not guarantee that the whole request becomes faster. The actual result depends on the model, runtime, hardware, batch size, memory layout and the amount of work done on each side of the call.
This approach is most attractive when a team already has a capable native inference runtime, needs to use its hardware support, and wants the surrounding service to remain in Java. It is also the more immediate path to investigate because FFM is already a finalized Java API. The project still has engineering work, but the API itself is not merely a Babylon experiment.
Babylon asks a different question: can Java code become the input?
Project Babylon's code reflection is not just reflection over objects. Traditional Java reflection can tell a program about classes, methods and fields. Code reflection aims to represent the structure of executable Java code so a library can inspect and transform it. The OpenJDK Babylon project describes its goal as extending Java to foreign programming models, including machine learning and GPUs.
That changes the shape of the integration. Instead of writing model behavior in one language and wrapping it for Java, a developer could express selected computations as Java code, let Babylon create a code model, and have a library lower that representation toward another execution engine. In the Devoxx session, Mihalceanu describes this future direction as expressing AI models as Java programs that Babylon can analyze, transform and lower to inference engines and accelerator backends.
The interesting promise is not simply “Java, but faster.” It is that Java could become a source language for multiple execution models. A library could use the same higher-level expression to target a CPU implementation, an accelerator, or another backend, if the code and backend support the required transformation. That is a much broader ambition than calling a native function.
But the conditional matters. The code must fit the subset the transformation understands. The target backend must support the operations. A library must bridge the difference between Java's semantics and the destination model. And developers need to inspect generated code, performance and correctness rather than assuming that a Java method automatically becomes an efficient GPU kernel.
The Heterogeneous Accelerator Toolkit, or HAT, is an example of this line of work. A separate Inside Java session on HAT describes using code reflection to turn Java compute code into a form that can be transformed for GPU execution, including OpenCL and CUDA backends. That is a concrete demonstration of the general idea, not proof that arbitrary Java code can be sent to a GPU.
The maturity caveat is essential. The OpenJDK Babylon repository says the API and implementation live in the incubating jdk.incubator.code module. This is active development, not a stable, drop-in feature that application teams should treat as a production contract. The project is valuable to follow precisely because it is exploring what could become possible, not because every Java AI application should adopt it today.
These tracks meet different needs
A useful way to choose is to start with what you are trying to keep stable.
- Keep Java as the application language, but reuse an existing native runtime: evaluate FFM and jextract. You still work with the runtime's model format and deployment requirements, but the interop layer can be more direct and Java-shaped.
- Keep model computation expressible in Java and explore alternative backends: follow Babylon and HAT. The potential payoff is a higher-level Java representation that can be analyzed or translated, but the APIs and supported transformations are experimental.
- Keep inference outside the application process: use a remote model service when independent scaling, managed hardware or operational separation matters more than in-process calls. Neither FFM nor Babylon makes a network boundary disappear.
- Keep the system easy to operate: compare total deployment and diagnostic cost, not only kernel speed. A native library can add platform-specific packaging; an experimental code-transformation stack can add toolchain and compatibility risk.
These are not mutually exclusive forever. A Java application could use a service boundary for some models, a native runtime for other workloads, and eventually a code-reflection library for selected compute-heavy functions. The right unit of decision is not “the whole company should choose one AI stack.” It is a workload with measurable requirements and a supportable deployment path.
What should a Java team test first?
Before choosing a path, write down the actual bottleneck. Is inference latency too high? Is model deployment awkward? Is a Python service difficult to operate? Is the team trying to use a particular accelerator? These problems can point to different solutions. A clean Java-to-native bridge will not fix a remote API's network latency. A GPU kernel will not fix an application that spends most of its time waiting for database calls.
For an FFM prototype, test a narrow slice. Load one runtime, run one model, send representative input, and measure both the native call and the full request path. Include model loading, input conversion, memory allocation, output conversion and shutdown. Check how the native library is packaged for every target operating system and architecture. Confirm that failure modes cross the boundary clearly enough for the service to report them.
For Babylon, keep the experiment separate from production code. Start from the project's own examples and identify which Java constructs the current toolchain supports. Compare the transformed result with a known-correct CPU implementation. Test numerical tolerance, resource ownership and backend behavior. Record the exact JDK build, Babylon revision, accelerator and driver, because results from an incubating project can change as APIs and transformations evolve.
In both cases, benchmark a realistic workload rather than a tiny arithmetic loop. A microbenchmark can show that a particular operation is fast; it cannot establish that a production request is faster or that a deployment is simpler. Track latency percentiles, throughput, memory use, startup cost and operational failure behavior. Keep the ordinary Java path available until the alternative has passed correctness and rollback tests.
The bigger idea is a change in the integration boundary
FFM and Babylon are easy to lump together as “Java calling native AI.” The more useful distinction is where the boundary sits. With FFM, the boundary is explicit: Java calls a native runtime and manages the interface. With Babylon, the ambition is to expose Java computation in a form a library can transform into a foreign model. One is primarily a safer, standardized bridge. The other is a new way for libraries to interpret Java code.
That difference explains why the two-track story is credible without pretending that Babylon has already replaced today's model runtimes. FFM is the practical bridge to investigate now. Babylon is the research and platform direction to watch if your work involves compute libraries, GPU backends or language-level integration. Their timelines and risks are different, and good architecture keeps that distinction visible.
The most useful takeaway is not that every Java team should move inference in-process. It is that “Java AI” can mean more than choosing a chat framework. It can mean deciding which runtime owns the model, how code reaches specialized hardware, and whether the application needs a stable interop API or a transformable program representation.
A short decision checklist:
- Name the workload and its measured bottleneck before choosing a technology.
- Prefer FFM when the goal is to call a mature native library from Java.
- Treat Babylon and HAT as experiments until their APIs and toolchains meet your team's support requirements.
- Measure the end-to-end request, not just the native call or accelerator kernel.
- Keep a correct fallback and document the native or incubating components you are adding.
I write about Java, Spring Boot, and AI every week. Subscribe if you want more practical breakdowns of how these technologies fit together.
Have you tried FFM with an inference runtime, or experimented with Babylon and HAT? Which boundary would you rather own in a production system?
Top comments (0)