I built Semwright to let AI agents work across desktop applications

Sergio Ramírez — Wed, 07 Oct 2026 22:25:19 +0000

For one of Semwright's demos, I started with an existing aircraft project in Blender. The workflow updates materials, adjusts the camera, renders a new shot, creates animated titles with Motion Canvas, and assembles everything with MLT.

The result is a finished video, along with an editable timeline. You can still change the timing, replace footage, adjust the audio, or continue working on the project afterward.

That's an important part of what I wanted to build. The goal isn't just to get an agent to produce an output, but to let it work with real software and leave behind something you can actually keep working on.

I've been building Semwright, an open-source runtime written in Rust that connects AI agents to desktop and professional applications.

Watch the 60-second demo

Working with the application itself

Take something simple, like asking an agent to change a material in Blender.

With screenshot-based automation, the agent has to find the right object, navigate the interface, locate the material settings, and interact with the controls.

But Blender already has an API for working with scenes, materials, cameras, and objects directly.

Semwright takes advantage of that.

It uses application-specific drivers to expose structured operations to agents. Instead of figuring out where to click, an agent can request an operation against the application's actual data model.

The runtime handles permissions, execution, and reading back application state where supported.

This means rearranging Blender's interface doesn't affect operations that use its API. And since the driver exposes a defined set of capabilities, the agent doesn't need unrestricted access to the application.

I've built integrations for Blender, Godot, Figma, LibreOffice, OBS, Motion Canvas, MLT, KiCad, and audio tools including Ardour and Faust. There's also a Chromium adapter.

The integrations have different scopes, but they follow the same underlying architecture.

What about MCP?

This is probably the easiest place to misunderstand what Semwright does.

MCP is a protocol that lets AI applications discover and interact with tools, resources, and other external capabilities. It's useful, and Semwright supports it.

But connecting an agent to a tool and managing what happens inside the software are two different problems.

For example, you could build a Blender MCP server that exposes a change_material tool. The agent can discover it and call it.

What happens behind that tool is up to its implementation.

How do you identify the correct object? What is the agent allowed to modify? How do you know the material actually changed? What happens to the resulting asset when another application needs it?

You can implement all of that in an MCP server. Semwright's approach is to provide those common mechanisms in one runtime, rather than rebuilding them separately for every integration.

With Semwright, MCP can be the interface the agent uses to send requests. Behind it, the runtime applies policy, dispatches operations through drivers, tracks references and artifacts, and checks observable results.

The same runtime can also be accessed through other interfaces, including the CLI.

So Semwright isn't an alternative to MCP. It's the execution layer behind the tools an agent uses.

Getting multiple applications to work together

This is where things started getting more interesting.

Blender understands its own scenes and objects. Motion Canvas handles animation. MLT works with video timelines.

Each application has its own data model, files, and way of doing things.

In the aircraft demo, the Blender output becomes an input to the video workflow. Motion Canvas generates the titles, and MLT assembles everything into the final project.

Semwright provides a common layer for those operations and artifact handoffs.

There are two parts of the architecture worth mentioning.

Project Graph keeps track of resource identities, relationships, and dependencies. It provides context about how different parts of a project relate to each other.

Effects helps distinguish between what an agent requested and what could actually be verified afterward.

For example, a command returning successfully doesn't necessarily mean the intended object was modified or the correct file was produced.

Being able to inspect the resulting application state matters, especially when the next operation depends on it.

The drivers handle application-specific details, while the runtime provides common behavior across them.

Why not just use the application's API directly?

You absolutely can. In fact, that's what many Semwright drivers do internally.

If you're writing a one-off Blender script, there's no reason to introduce another runtime just to change a material.

But when you start building agent workflows across several applications, you end up dealing with many of the same problems repeatedly: permissions, application state, references, dependencies, artifact transfers, and checking what actually happened.

If you switch agents or add another application, you shouldn't have to rebuild all of that infrastructure.

That's the idea behind Semwright.

You integrate an application through a driver, and compatible agents can use its supported capabilities through the same runtime and authorization model.

The repository also includes a Driver SDK for developing integrations and a Native SDK for applications that want deeper cooperation with their own state and transactions.

Open source

Semwright v1.0.0 is now available, along with the source code, documentation, SDKs, and demos.

I'm continuing to expand the application integrations and work on more advanced cross-app workflows, including preserving existing project state and improving recovery when part of a workflow fails.

There's a lot of interesting work in making agents useful beyond a single application, particularly when the output needs to remain editable and usable by someone afterward.

You can explore the project here:

GitHub — seradotcom/semwright

I'd be interested to hear what applications people are currently automating, especially if you're connecting several of them together. I'm also curious how others are handling shared state and verification across their agent tools.