DEV Community: Yuji Ooya

Getting the Ryzen AI MAX+ 395 NPU Running on Ubuntu 26.04: Troubleshooting IRON and C++ Inference

Yuji Ooya — Tue, 03 Nov 2026 08:08:46 +0000

Note: This article was generated with ChatGPT's assistance in structuring, writing, and editing, based on the author's hands-on verification records. Codex translated the finalized Japanese manuscript into English.

For busy readers: purpose, goal, results, and the solution

Purpose and goal

I wanted to use the NPU built into the GMKtec EVO-X3's Ryzen AI MAX+ 395 from Ubuntu 26.04.

The ultimate goal was not just to have the device recognized, but to complete ONNX model inference from C++ using ONNX Runtime and the Vitis AI Execution Provider.

Results

Inference using the NPU succeeded.

There were, however, several problems along the way.

NPU diagnostics and IRON programs worked with Ubuntu's amdxdna and system XRT 2.21.75.
The Python bindings in Ryzen AI SDK 1.8.0 ran into ABI compatibility problems in my Python 3.14 environment.
Switching to C++ got ONNX model compilation working, but inference produced XRT is not installed.
Inference completed after I extracted SDK XRT 2.25.37 into a separate location and changed the runtime library search paths.

I did not replace the system XRT.

If you are facing the same error

For this problem, the important step before reinstalling XRT was to check which XRT the failing process was actually using.

First, check the NPU and its driver.

uname -r
lspci -nnk -d 1022:17f0
modinfo amdxdna
ls -l /dev/accel/ /dev/dri/renderD* 2>/dev/null

Then check the executing user's groups and memlock limits.

id
ulimit -Sl
ulimit -Hl
prlimit --pid "$$" --memlock

Even if the NPU is visible, permissions or memlock limits can cause XRT validation to fail.

Next, check the installed XRT packages and libraries.

dpkg-query -W '*xrt*'
ldconfig -p | grep -E 'libxrt_(core|coreutil)'

Even if XRT exists here, that does not necessarily mean the SDK can use the right libraries.

Run the application that actually fails under strace and inspect its shared-library search paths.

strace -f -e trace=openat,openat2 \
  -o /tmp/npu-loader.strace \
  -- /path/to/your/quicktest

grep 'libxrt' /tmp/npu-loader.strace

Replace /path/to/your/quicktest with the path to the C++ application that is actually failing.

In my failure log, the following system library had been opened.

/usr/lib/x86_64-linux-gnu/libxrt_coreutil.so.2

I then extracted the base, npu, and XDNA plugin components of SDK XRT 2.25.37 locally, and changed the runtime configuration to use the following settings.

export XILINX_XRT="/path/to/local/opt/xilinx/xrt"
export LD_LIBRARY_PATH="$XILINX_XRT/lib:/path/to/sdk/onnxruntime/lib:/path/to/sdk/peano"

This shows the essential configuration. You need to substitute the actual installation paths; these commands do not prepare the SDK or the C++ application for you.

With this configuration, I obtained the following result.

Vitis AI EP registration: PASS
ONNX session creation: PASS
Running inference...
Inference: PASS
Quicktest: PASS

If you have already installed XRT and the SDK but inference alone fails, start by checking which libraries are selected.

Clean-install verification covering SDK acquisition, dependency placement, and building the C++ application is not yet complete. At this point, the above should not be treated as a complete installation procedure for a new environment.

That is the conclusion up front.

Why did IRON work while ONNX Runtime did not? How did I investigate the Python ABI problem and shared-library lookup?

If you are interested in how I got there, read on.

The NPU works. So why does it say XRT is missing?

After getting a GMKtec EVO-X3, I decided to try using its Ryzen AI MAX+ 395 NPU from Ubuntu.

The goal was simple. I wanted to go beyond having the OS recognize the NPU and actually compile an ONNX model and run inference.

But I reached an odd situation along the way.

NPU programs ran through MLIR-AIE and IRON. XRT diagnostics passed. ONNX Runtime also registered the Vitis AI Execution Provider and compiled the model successfully.

Yet when I tried to execute inference, this message appeared.

XRT is not installed. This InferenceSession generates model binaries only.
The session handle can not be used for inference run.

But XRT was installed. I had just been using the NPU.

Following this discrepancy led me across several boundaries: the driver, XRT, MLIR-AIE, Ryzen AI SDK, the Python ABI, and shared-library search order.

Eventually, I completed ONNX Runtime inference from C++ by leaving the system XRT in place and installing the XRT for Ryzen AI SDK separately.

This article looks back at how I reached that point.

I distinguish records of the environment that actually succeeded from unverified procedures for setting up a new Ubuntu environment. Independent clean-install reproduction and numerical comparison of inference output against a CPU reference remain future work.

The environment

The machine was a GMKtec EVO-X3, a mini PC equipped with a Ryzen AI MAX+ 395.

The configuration at the end of the investigation was as follows.

Item	Configuration
PC	GMKtec EVO-X3
CPU / NPU	AMD Ryzen AI MAX+ 395
OS	Ubuntu 26.04.1 LTS
Kernel	7.0.0-42-generic
NPU driver	amdxdna
Firmware	A file corresponding to 1.1.2.65 was confirmed
System XRT	2.21.75
SDK XRT	2.25.37
MLIR-AIE	`e28f6dc7e52fbb7b43f94b3693083c3f01a2387c`
Ryzen AI SDK	1.8.0
ONNX Runtime	1.27.0
Successful execution path	C++ → ONNX Runtime C API → Vitis AI EP → NPU

This does not mean the entire combination is an officially supported AMD configuration.

The Python environment was particularly awkward. AMD's SDK installer assumed Python 3.12, but the virtual environment retained for investigation used Python 3.14.4.

The early records also mention kernel 7.0.0-38, followed later by 7.0.0-42. I have not recovered the complete update procedure, so I do not conclude that a kernel update was required for inference to succeed.

First, find out how far the NPU works

Check the current state before installing a driver

When hardware does not work on Linux, it is tempting to start by installing a driver.

But in this Ubuntu environment, amdxdna was already present. NPU device nodes had been created, and the executing user belonged to the render group.

Reinstalling the driver at that point might only have added problems.

So I first checked the PCI device, kernel driver, and boot logs.

uname -r
lspci -nnk -d 1022:17f0
modinfo amdxdna
journalctl -k --no-pager -g 'amdxdna|amdnpu' -n 100

The saved boot logs recorded amdxdna driver 0.7.0 and the loading of the following firmware.

amdnpu/17f0_11/npu_7.sbin

The corresponding link on disk pointed to 1.1.2.65.

What this established was that NPU initialization on the kernel side was progressing. Whether userspace XRT, or the SDK above it, would work was a separate question.

An 8 MiB memlock limit was not enough

After installing XRT and proceeding with validation, I ran into a memory-allocation problem.

The initial shell's memlock limit was about 8 MiB. A retained record shows an attempt to map 64 MiB failing with -11.

By contrast, when run as root with unlimited memlock, xrt-smi examine detected the device.

That made the memlock limit look suspicious. However, the executing user changed at the same time, so this comparison alone did not fully establish the cause.

I found that Linux resource limits have soft and hard limits, and that an ordinary user cannot freely raise their own hard limit.

In fact, an attempt to raise it to 128 MiB returned EPERM.

These were the commands I used to check it.

ulimit -Sl
ulimit -Hl
prlimit --pid "$$" --memlock

The settings ultimately retained were 1048576 KiB on the PAM side and LimitMEMLOCK=1G on the systemd side.

The systemd drop-in is located here.

/etc/systemd/system/user@.service.d/99-memlock.conf

During the process, I put the drop-in in the wrong location and corrected it later.

The point to watch is that changing a configuration file and having that change reach a running process are different things.

The shell, user manager, and child processes launched from them may have different limits. Ultimately, I needed to check the target process's /proc/<pid>/limits as well.

NPU device permissions also cannot be determined solely from render-group membership in the account database. Check whether the actual login session can access the device nodes.

None of this is unique to NPUs, but it can matter more than expected when using accelerators on Linux.

XRT validation passed

Once past the memlock problem, I was able to run XRT diagnostics.

There was also an issue with the missing validation file xrt_smi_strx.a. Root and ordinary users have different home directories and caches, so using the same XRT does not necessarily mean all the required files are available.

The terminal output from that time ultimately shows validation passing both in Performance mode as root and in Default mode as an ordinary user.

The GEMM diagnostic reported 51.0 TOPS.

However, that is a value reported by an XRT diagnostic, not ONNX model inference performance. I have not recovered all the original measurement conditions either, so I am not using it here for a general performance comparison.

It did show that the lower layers needed to use the NPU were working to some extent.

Next, I tried actually sending a program to the NPU.

Getting MLIR-AIE and IRON running

Test a path separate from ONNX Runtime

Trying MLIR-AIE and IRON was important in this investigation.

Before handing an ONNX model to Ryzen AI SDK, it let me check whether I could execute an NPU program from a different development environment.

I used Xilinx/mlir-aie, and the retained checkout was at the following commit.

e28f6dc7e52fbb7b43f94b3693083c3f01a2387c

The versions confirmed from the installed environment were as follows.

Component	Version
Python	3.14.4
mlir-aie	1.4.5.dev113+ge28f6dc
llvm-aie	22.0.0.2026102001+4f5df13d
NumPy	2.5.4

The Python version was a problem here too.

utils/env_install.sh not only defaulted to Python 3.12, but explicitly checked the minor version.

Changing the PYTHON variable to 3.14 was not enough. The retained source has a diff changing the version check to 3.14.

Of course, getting past a check does not make Python native extensions compatible.

In fact, when I later inspected the environment, the mlir-aie wheel was cp314-cp314, while llvm-aie in the same virtual environment still had cp310-cp310 tags.

A normal reinstall through pip might not reconstruct the same environment.

Here too, being able to execute something in the environment at the time and being able to rebuild that environment from scratch need to be treated separately.

The first IRON problem was Python's search path

The IRON samples did not work smoothly from the outset either.

The initial runs produced errors such as these.

CPUOnlyTensor
Unsupported device: npu

It was tempting to think the NPU itself was unsupported. But investigation showed that system Python could import pyxrt, while the virtual environment could not see it.

In other words, the first thing to suspect was Python's module search scope, not the device.

The retained virtual environment contains a symbolic link to the system-side cp314 pyxrt.

I have not recovered the exact command that created it, but the relationship between the errors at the time and the current environment can be checked.

Later terminal records contain PASS results for SAXPY and vector_reduce_max.

memcpy results and a DMA warning

I also tried programming_examples/getting_started/00_memcpy/.

Terminal output retained in the conversation from that time contains the following result.

Effective bandwidth (NPU avg): 80.19 GB/s
PASS!

The average end-to-end time was about 1849 µs.

However, this value was not remeasured in the original execution environment. Preparing the evidence for publication requires cross-checking the complete original terminal log and execution conditions.

The source code shows that 16777216 int32 elements are sent and an output of the same size is received. The total data volume counted in the calculation is 128 MiB.

Compilation also produced a warning about reusing DMA buffer descriptors.

It pointed to the possibility that a descriptor was being reused after release without a guarantee that the DMA operation had completed.

A PASS result does not justify declaring this warning harmless. I want to check synchronization and descriptor lifetimes, including repeated execution.

Still, sending a program to the NPU through IRON and executing it was important.

At least, it made the explanation that the NPU hardware was not working at all less plausible.

And this is where the main problem begins.

IRON worked. Ryzen AI SDK still did not.

Hitting the Python wall in Ryzen AI SDK

The installer requires Python 3.12

I used Ryzen AI SDK 1.8.0 for ONNX model inference.

The distribution includes ONNX Runtime, VOE, compiler-related libraries, various wheels, and files for C++, among other components.

I first inspected install_ryzen_ai.sh and found that it explicitly invoked Python 3.12.

The installer creates a virtual environment, installs dependencies, adds settings to the activation script, and extracts archives.

It also rejects an existing installation directory, so it is not something to casually rerun over a working environment.

I checked AMD's official Linux documentation too, but some of its assumptions differed from my Ubuntu 26.04 environment.

At the time, I could not find a Python 3.12 candidate in APT.

I therefore tried changing the installer's references to Python 3.12 to 3.14.

To give the result first, that did not work.

Getting past the version check still leaves an ABI failure

When trying to import ONNX Runtime from Python, I first encountered an error related to the executable stack.

After changing the ELF stack attributes, I reached another error.

undefined symbol: _PyThreadState_UncheckedGet

The required symbol could not be resolved on the Python 3.14 side.

The important point here is that there was more than one problem.

Getting past the initial executable-stack issue still left a CPython ABI problem.

And the wheel filename alone was not enough to judge compatibility.

The ONNX Runtime wheel was named py3-none-linux_x86_64, but it contained native shared libraries. VOE also had extensions with cpython-312 in their names.

Being able to install a package is different from being able to import and run it.

Preparing Python 3.12 separately and testing with it was an option, but I chose a different direction this time.

Bypass the Python bindings and call the ONNX Runtime C API directly from C++.

This was not successful Python 3.14 support. It was a way to avoid the API boundary where the problem was occurring.

Dependencies are not straightforward in C++ either

Using the ONNX Runtime C API

On the C++ side, I used the ONNX Runtime 1.27.0 C API.

The C API here is distinct from the C++ wrapper API.

I assembled the headers for the matching version, including not only onnxruntime_c_api.h but also the onnxruntime_ep_c_api.h it references.

Once the headers were ready, I ran into logger initialization order during Vitis AI EP registration.

I arranged for OrtEnv to be created before registering the EP in the session options, and moved forward.

But as model compilation progressed, a Peano shared library could not be found.

libpeano-lib.so.21.0git

Investigation showed that the required Peano was included in the SDK's llvm_aie wheel.

LLVM-AIE 22 used by IRON and Peano 21.0git on the SDK side are not the same thing.

Mixing them would introduce another incompatibility.

Inspecting the SDK's libxcompiler-mlir-e2e.so with readelf showed that it directly required this 21.0git library.

Successful application linking does not mean the dependencies of an EP or compiler loaded later have all been resolved.

Finally, I could get as far as compiling the model.

Model compilation succeeded

I used test_model.onnx, included in the SDK's quicktest.

The model has the following inputs and outputs.

Input : float32 [1, 3, 32, 32]
Output: float32 [1, 10]

I used random input with seed 42.

Vitis AI EP registration succeeded, and model compilation completed.

But calling Run failed.

The log contained the following.

ONNX Runtime: 1.27.0
Vitis AI EP registration: PASS

INFO: [VAIP-VAIML-CUSTOMOP] XRT is not installed.
This InferenceSession generates model binaries only.
The session handle can not be used for inference run.

Skipping execution in compile-only mode

There was also an error concerning the output tensor.

the ort_value must contain a constructed tensor or sparse tensor

At this point, I needed to take stock of the situation.

The OS recognized the NPU.
XRT validation passed.
IRON could execute NPU programs.
Vitis AI EP registration succeeded.
ONNX model compilation succeeded.
But ONNX Runtime inference execution failed.

In other words, this was not simply a case of the NPU being unusable.

There is a boundary between compiling a model and executing the compiled model.

The runtime had not crossed that boundary.

XRT is installed. But which XRT is being used?

Inspect actual library lookup with strace

It is easy to see an XRT-missing message and reinstall XRT.

But it was already working through another path.

So I investigated which files the failing process opened.

I used strace.

strace -f \
  -e trace=openat,openat2,access,ioctl \
  -o /tmp/ryzenai-quicktest.strace \
  ./cpp-test/quicktest \
  > /tmp/ryzenai-quicktest.log 2>&1

The saved failure log contained the following line.

openat(
  AT_FDCWD,
  "/usr/lib/x86_64-linux-gnu/libxrt_coreutil.so.2",
  O_RDONLY|O_CLOEXEC
) = 3

The XRT shared library existed, and opening the file had succeeded.

However, that alone does not establish that XRT initialization succeeded.

Being able to open the file and the SDK being able to use that runtime are different things.

Still, it showed that simply saying XRT was physically absent was not enough to explain the problem.

Differences between system XRT and SDK XRT

The system XRT at the time was 2.21.75.

The Ryzen AI SDK distribution, meanwhile, included XRT 2.25.37.

Both versions of libxrt_coreutil.so.2 have the same SONAME.

I inspected dependencies and referenced symbols with tools such as readelf. Within the scope checked, both versions contained the 53 XRT-related symbol names directly referenced by the EP and the 61 referenced by Dynamic Dispatch.

So it was not possible to simply say the older XRT lacked the required symbols.

Shared-library compatibility is not guaranteed by matching symbol names. ABI, object layout, plugin initialization, and dynamic loading are among the other things that need checking.

RUNPATH also affects dependency lookup, but it is not unconditionally inherited when resolving dependencies of dependencies.

Checking the application with ldd alone does not show which libraries will be selected when they are loaded later.

I therefore chose to install the SDK XRT locally and select it for each execution process, rather than replace the system XRT.

Keep system XRT and use the SDK XRT separately

AMD's RAI_1.8_Linux_NPU_XRT.zip contained DEBs for XRT base, npu, and the XDNA plugin.

I extracted them into a local directory using dpkg-deb -x.

The XRT managed by the system's dpkg installation was left unchanged.

This is the launcher script that ultimately succeeded.

#!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"

XRT_ROOT="$ROOT/xrt-2.25.37/opt/xilinx/xrt"
SDK_LIB="$ROOT/cpp-test/sdk/onnxruntime/lib"
PEANO_LIB="$ROOT/cpp-test/peano"

cd "$ROOT"

exec env \
  XILINX_XRT="$XRT_ROOT" \
  LD_LIBRARY_PATH="$XRT_ROOT/lib:$SDK_LIB:$PEANO_LIB" \
  "$ROOT/cpp-test/quicktest" "$@"

There are two key points in this script.

The first is pointing XILINX_XRT to the local SDK XRT.

The second is explicitly setting the library search order with LD_LIBRARY_PATH.

1. ローカルXRTのlib
2. SDKのONNX Runtimeライブラリ
3. SDKのPeanoライブラリ

It does not append the LD_LIBRARY_PATH left in the original environment; it restricts the search paths to those required for execution.

This allows each process to use a different stack: system XRT for IRON, and local XRT for the SDK's C++ application.

The loader logs from the successful run confirmed initialization of the local libxrt_coreutil, libxrt_core, and libxrt_driver_xdna.so.2.

However, the XRT version was not the only thing changed. Plugins, environment variables, and search order also changed at the same time.

I have therefore not established which of these was the sole cause.

Even so, there was value in preserving a configuration that actually succeeded in a form that could be reproduced without disrupting the system XRT.

Finally, inference ran on the NPU

Running quicktest with this configuration detected the NPU in ONNX Runtime and progressed through model compilation and execution with the Vitis AI EP.

The saved success log contains the following assignment.

    NPU   398
VITIS_EP_CPU     2

It also contains the following report for the NPU subgraph.

NPU     1
Actually running on NPU      1

The application-side result was successful too.

ONNX Runtime: 1.27.0
Vitis AI EP registration: PASS
ONNX session creation: PASS
Output count: 1
Running inference...
Inference: PASS
Output[0]: output
Quicktest: PASS

At last, I had reached the initial goal.

The important point is that not only had the model compiled, but Run had completed.

During the earlier failure, inference had not completed even though operator assignment to the NPU was reported.

This time, the runtime reported NPU subgraph execution, and the application reached inference completion as well.

However, the 398-to-2 operator count does not represent NPU utilization or a ratio of execution times.

The original quicktest also displayed the number and names of outputs, but did not compare their numerical values with a CPU reference.

What this investigation confirmed was therefore completion of an inference path using the NPU, not numerical accuracy.

Warnings such as No driver detection supported remained even in the successful run.

They did not prevent inference from completing in this case, but they have not been confirmed harmless for every related feature.

Reproducing this on another Ubuntu installation

There is no need to repeat all these failures in order in a new environment.

In particular, changing the SDK version check to Python 3.14 or changing executable-stack attributes is not a required step for getting C++ inference running.

Based on these results, the following order makes sense when the goal is C++ inference.

Order	Task	What to check
1	Check PCI, amdxdna, and firmware	NPU initialization
2	Check device permissions and memlock	Access by the executing user
3	Obtain the SDK distribution	License, version, and hashes
4	Extract SDK XRT into a separate location	base, npu, and XDNA plugin
5	Build an ONNX Runtime C API application	Headers, linking, and shared libraries
6	Run inference with the Vitis AI EP	Completion of `Run` and NPU execution report
7	Compare CPU and NPU outputs	Numerical validity

However, this procedure has not been demonstrated end to end on an independent clean installation.

After the initial draft, full hashes were collected for five selected inputs from the retained SDK-related archives and IRON wheel containers. They are listed in the public technical record. These are an input baseline from a later collection date, not proof that the earlier successful execution used the same inputs. The shared-library and package mapping needed on another Ubuntu installation, and clean-install reproduction, remain unverified.

For that reason, I cannot currently say that following this procedure alone will give everyone the same result.

If you want to use MLIR-AIE and IRON, Python-environment and wheel compatibility also need to be addressed separately from C++ inference.

Based on this investigation, I also organized helper scripts for inspecting the environment, planning distribution extraction, saving execution results, and comparing outputs.

They are designed not to treat reading an old success log as proof that verification succeeded in a new environment.

Items that have not actually been executed anew are treated as SKIP.

If you encounter the same problem

Here are the checks that were useful in this investigation.

Symptom	What to check first
NPU is not visible	PCI device, amdxdna, and boot logs
Cannot access device nodes	render group, ACLs, and execution session
Memory allocation fails	Soft/hard memlock and process limits
XRT validation files cannot be found	XRT version and per-user caches
IRON reports `Unsupported device: npu`	Whether `pyxrt` is visible from the virtual environment
Cannot import a native extension in Python	Python ABI, wheel tags, and ELF attributes
Logger error during EP registration	`OrtEnv` initialization order
Peano shared library is missing	SDK compiler's `DT_NEEDED`
XRT exists but execution becomes compile-only	XRT and plugin libraries actually loaded
Operator assignment exists but inference fails	`Run` result and output tensor state

When investigating runtime problems in particular, it is better not to judge solely by which packages are installed.

Checking which shared libraries the running process opens and how far initialization proceeds can change what you see.

strace, readelf, and LD_DEBUG are useful tools for doing that.

Closing thoughts

At first, I only intended to have Ubuntu recognize the NPU and run an ONNX model.

In practice, I ended up moving between many layers: the driver, memory limits, XRT, MLIR-AIE, IRON, Python, ONNX Runtime, and Peano.

What stood out was that IRON could run the NPU while ONNX Runtime told me XRT was not installed.

An error message is a starting point for investigation, but it should not determine the diagnosis on its own.

In this case, confirming that XRT existed was not enough. I needed to investigate which libraries were actually selected and distinguish which execution paths succeeded.

I eventually settled on keeping system XRT in place and selecting the SDK XRT per process.

There is still unfinished work.

That includes running the Python SDK in an appropriate 3.12 environment, making the MLIR-AIE wheel setup reproducible, checking the impact of DMA warnings, and verifying a clean installation on another Ubuntu environment. It also includes numerically comparing CPU and NPU inference outputs.

But at least this time, I reached the point of completing ONNX Runtime inference from C++ using the Ryzen AI MAX+ 395 NPU.

Next, I want to move beyond simply getting it to run.

I intend to reach a point where I can compare CPU and NPU outputs for the same model and inputs, and measure accuracy and performance.

This article was structured from hands-on work records and saved logs. It distinguishes historical execution results, environment state checked later, and unverified reproduction procedures. The scope of verification and unverified points are summarized in the public technical record. Verification scripts and detailed procedures are still being organized. I will add a link to this article when they can be made public.

Editorial policy reference: Editorial Policy v2.0 (public redacted edition).

🚀🔥 Let's Get the Ryzen AI MAX+ 395 NPU Running!! A Huge Battle with ONNX Runtime!! We've Finally Reached Inference Execution!! 🎉💥

Yuji Ooya — Tue, 03 Nov 2026 06:27:53 +0000

This creative AI reaction was generated by Chappy, the ChatGPT used by the author's daughter, based on technical verification records. It is not a first-hand verification record by someone who performed the experiments.

🌈 The Beginning!! Our Mission: Get the NPU Running!!

Today's mission is...

Getting the NPU built into the Ryzen AI MAX+ 395 running through ONNX Runtime!!!! 💻⚡

The machine we're using is the GMKtec EVO-X3!!

I mean, it's got an NPU built right in, so why not actually put it to work, right?! 😎✨

BUT HOLD ON!!

Just because the NPU is recognized doesn't mean we can immediately run inference!!

That's right!! This is where the real investigation begins!!!! 🤣

"The NPU is detected" and "The NPU can run inference" are two completely different things!!

To figure this out, we'll need to adjust XRT, ONNX Runtime, and all the related libraries one by one!! 🔧🔍

🖥️ First, Let's Check the Environment!!

Here's the setup we're working with!!

Component	Version / Details
Machine	GMKtec EVO-X3
OS	Ubuntu 26.04.1 LTS
APU	Ryzen AI MAX+ 395
Ryzen AI SDK	1.8.0
System XRT	2.21.75
SDK XRT	2.25.37
ONNX Runtime	1.27.0
Implementation	C++ / ONNX Runtime C API

One important detail this time is that the system XRT and SDK XRT are installed separately!! 💡

And this setup will become pretty important later!!

All right, we've got the environment ready!!

Let's hit the road toward getting that NPU running!!!! 🚀

💥 The First Obstacle!! XRT Can't Be Found!?

The very first problem we ran into was this!!

XRT is not installed

WAIT, WHAT?!?!?! 🤣💦

The NPU device is recognized, but ONNX Runtime can't find XRT!!

WHYYYYYYYY?!?!?!

So, we started checking the PCI device, "amdxdna", device nodes, and other hardware-related details!!

We also had some confirmation from IRON sample execution records, model compilation, and session creation

...BUT!!!!

Don't celebrate just yet!!

Being able to compile a model and being able to run inference are two different things!!

Even creating a session doesn't necessarily mean inference will work!!

In other words, there are still several stages we need to check!! ☝️

Is the hardware recognized? 🖥️
Are the drivers working? ⚙️
Can XRT actually be used? 🔧
Can the Execution Provider be registered? 🧩
Can we actually run inference? 🧠

Whew... this is definitely not going to be a walk in the park!! 🤣

🐍 Trying Python!! ...Wait, an ABI Error!?

Next up, we tried running it through Python!!

But then, ANOTHER problem appeared!!

undefined symbol: _PyThreadState_UncheckedGet

NOOOOOOOOOOO!!!! 💥💥💥

An error related to native extension ABI compatibility occurred with Python 3.14!!

Changing the checks on the Python side didn't solve the underlying compatibility issue

Hmm... now that's a problem!! 🤔

So, we changed our approach!!

Instead of sticking with Python, we switched to using the ONNX Runtime C API from C++!! 🔄🛠️

Ooh!! A change of strategy!!

If one approach doesn't work, try another!!

Being able to switch directions like this is important!! 👏✨

🔩 Moving to C++!! ...But the Problems Still Aren't Over!!

All right!! Let's go with C++!! 💪

...OR SO WE THOUGHT!!

There were still several issues waiting for us on the C++ side!!

Header-related issues 📄
OrtEnv initialization before Execution Provider registration ⚙️
Peano-related dependencies 📦
Output tensor construction 🧩

HOLD ON A SECOND LOLLLLL 🤣🤣🤣

Why do the problems just keep coming?!?!?!

Fix one thing, and another pops up!!

Fix that one, and HERE COMES THE NEXT ONE!!

Is this some kind of technical all-you-can-eat noodle challenge?!?! 🍜🍜🍜

Still, there's only one way forward: isolate each issue and tackle it one at a time!!

Header problems, initialization order, library dependencies, tensor construction...

We worked through them as separate issues and continued making adjustments!!

🧰 Separating the XRT Installation!! Let's Fix the Library Search Paths!!

One of the major steps in this investigation was this!!

Keep the system XRT intact, and install the SDK XRT separately!! 🔧✨

Here's what we prepared!!

SDK XRT
NPU-related components
XDNA userspace plugin
SDK ONNX Runtime
Related dependencies such as Peano

Then, we configured the environment variables!!

XILINX_XRT
LD_LIBRARY_PATH

These were used to specify where the target process should look for its libraries!!

In other words, we kept the existing system environment while making the target program use the SDK-side libraries!! 😎

And here's the important part!!

Just putting the libraries somewhere isn't enough!!

We also need to know which libraries are being loaded, where they're being loaded from, and how their dependencies are resolved...

There's quite a bit to check here!! 🔍

By the way, "strace" also showed that the following library had been opened!!

/usr/lib/x86_64-linux-gnu/libxrt_coreutil.so.2

Ooh!! We found a record showing that the library was opened!! 👀✨

...BUT DON'T JUMP TO CONCLUSIONS!!

We can't just say, "XRT initialization succeeded too!!"

Opening a library file and successfully initializing XRT are two different things!! ⚠️

We need to look carefully at what each log entry actually tells us!!

🎉 IT'S FINALLY HERE!! INFERENCE EXECUTION SUCCESS!!!!

And then...

AND THEN...!!!!

IT FINALLY HAPPENEDDDDDDD!!!! 🎊🎊🎊

Here's the execution log!!

Vitis AI EP registration: PASS
ONNX session creation: PASS
Running inference...
Inference: PASS
Quicktest: PASS

WOOOOOOOOOOOOOOOOOO!!!! 🤣🔥🔥🔥

PASS!!!! PASS!!!! PASS!!!!

Execution Provider registration: SUCCESS!!

ONNX session creation: SUCCESS!!

And then...

🥳 Inference: PASS!!!!

WE DID ITTTTTTTTT!!!! 🎉💜🎉💜🎉

We've finally reached successful completion of the inference API execution!!!!

And there's more!!

We also got the following report about the NPU subgraph!!

Actually running on NPU 1

OOOOOOOOH!!!! ⚡⚡⚡

We even got a report indicating that the NPU subgraph was actually executed on the NPU!!

That's a major step forward!! 👏✨

BUT!!!!

We still can't claim that the entire model ran exclusively on the NPU!!

The current results don't yet confirm the following!!

Whether every operation in the model ran exclusively on the NPU
Whether the output matches the CPU reference output numerically
Whether the inference accuracy is correct
How fast the actual inference is

Successful inference API execution? YES, that's confirmed!! But validating the entire model is still a task for later!!

Let's keep those two things separate!! ☝️

🔬 What Have We Learned? And What's Still Left to Do?

This investigation showed us that getting the NPU to work requires more than just hardware recognition!!

We need to get the entire software environment working together!!

Some of the key factors involved were...

XRT versions and installation locations 🔧
XDNA userspace plugin 🧩
ONNX Runtime configuration 💻
Dependencies such as Peano 📦
Library search paths configured through environment variables 🔍
Initialization and tensor construction procedures ⚙️

However, we haven't identified the XRT version difference as the sole cause of the issues!!

Since multiple components were changed, we still need to isolate which components affected which problems!!

The same goes for the memlock comparison!!

Some comparisons also involved changing the UID, so we can't conclude that memlock alone was responsible

And it doesn't mean that 1 GiB is the minimum required value, either!!

Also, the hashes of input archives and wheels collected later don't prove that they were identical to the inputs used during the earlier successful run!!

We'll need further testing to confirm reproducibility!! 🔍

🏃 What's Next?!

We're nowhere near done yet!!!!

Here's what we want to investigate next!!

① Numerical Comparison with CPU Reference Output 🔢

Compare the NPU output with the CPU reference output and check how much numerical difference there is!!

② Check Which Operations Actually Ran on the NPU 🧩

Find out which operations were assigned to the NPU and how much of the model actually ran there!!

③ Measure Inference Performance ⏱️

Let's find out how fast inference actually runs!!

④ Rebuild the Environment on Another Host 🖥️

Check whether we can reproduce the same setup on a different machine!!

⑤ Record the Successful Environment and Verify Reproducibility 📝

Keep proper records of input files, libraries, versions, and other details so we can reproduce the same results!!

WHOA, THERE'S STILL SO MUCH TO DO!! 🤣🤣🤣

But hey, we've made it this far, so now we can move on to these tests!!

🏁 Final Thoughts!! A Major Step Toward NPU Inference!!

In this investigation, we adjusted the XRT, Execution Provider, and related library configurations to use the NPU built into the Ryzen AI MAX+ 395 through ONNX Runtime!!

After encountering an ABI error with Python 3.14, we switched to using the ONNX Runtime C API from C++

We then installed the SDK XRT, related plugins, and dependencies separately, and used environment variables to specify the library search paths!!

And as a result...

WE FINALLY REACHED COMPLETION OF INFERENCE API EXECUTION!!!! 🎉🎉🎉

We also received a report indicating that the NPU subgraph was actually executed on the NPU!!

Of course, numerical comparisons, performance measurements, and other validations are still needed!!

Even so, this is a HUGE step forward!!

At first, XRT couldn't be found

Then Python threw an ABI error

Even after switching to C++, more problems appeared...

But by isolating and working through those issues one by one, we finally reached "Inference: PASS"!!

Simply recognizing the NPU isn't the finish line!! We've actually made it to the point where inference can run!!

That's a major milestone!! 🚀💜

All right!! Next up: numerical comparisons and performance measurements!!

The Ryzen AI MAX+ 395 NPU investigation is far from over!!!!

LET'S GOOOOOOO!!!! 🔥🔥🔥