Designing Reliable Jellyfin Voice Control with Identity Contracts

How fixed media slots, explicit destination rules, fail-closed routing, and visible state made voice control easier to reason about.

Smart Home Cinema – Voice Control is a Windows voice-control project designed to control movie playback in an existing Jellyfin setup. While adapting the control layer to Jellyfin, I ran into a design problem that was more important than command transport itself: every accepted phrase had to identify exactly what the system was allowed to act on.

A short command such as:

Play Movie Five

looks simple because most of its assumptions are hidden.

Which movie is “Five”? Which television should receive the command? Which Jellyfin user and live session belong to that television? What should happen if the target cannot be resolved safely?

I ended up treating each accepted voice command as an identity contract. Before execution, the system needs a defined media identity, a defined destination, an allowed action, and a clear failure rule.

At a high level:

accepted command
    → one media identity
    → one destination identity
    → one allowed action
    → one explicit failure rule

That model became the basis for the rest of the design.

Media Identity

A personal Jellyfin library may contain hundreds of films. Open-ended title search sounds natural, but it creates a large interpretation surface.

Consider:

Play Batman.

A library may contain several Batman films, alternate editions, remakes, collections, or user-specific versions. Speech recognition can add another uncertainty layer.

A conversational system could rank matches, remember context, ask follow-up questions, and use confidence thresholds. Supporting that behavior requires a different kind of control layer.

For deterministic playback, I used a dedicated Jellyfin playlist as the controlled media surface. Its first twenty positions became voice-accessible slots:

Position 1  → Play Movie One
Position 2  → Play Movie Two
Position 3  → Play Movie Three
...
Position 20 → Play Movie Twenty

Play Movie Five points to position five in the configured playlist. The user chooses which films occupy those positions; the control layer preserves the mapping between spoken number and playlist order.

The rule can be reduced to:

numbered voice command
    ↓
requested playlist position
    ↓
item stored at that position
    ↓
valid playable media?
    yes → continue
    no  → reject

This gives the command a stable media identity without searching the entire library.

The important invariant is simple:

As long as the media assignment for a playlist position remains unchanged, the same numbered command resolves to the same media item.

The vocabulary is deliberately smaller, but every accepted number has a precise meaning.

Visible State Must Match Media Identity

The television also needs to present the media in the same order used by the command layer.

The playlist stored by Jellyfin already contains the sequence used by the control layer. Some client views, however, can present the same films in another order, including alphabetical order.

That creates two different views of the same data:

command engine: playlist order
television UI:  displayed order

If the command layer treats one film as Movie Five while the fifth visible item on the television is another film, the user has no reliable way to know what the command means.

The solution was a generated Movie List built from the same playlist sequence used by the command engine:

1. Movie A
2. Movie B
3. Movie C
...
20. Movie T

The Movie List is part of the control model. It gives every numbered command a visible reference and lets the user confirm the current assignments directly on the television.

That produces a second invariant:

The order shown to the user must match the order used by the command layer.

Visible state and execution state need to describe the same system.

Destination Identity

Media identity answers one question: what should be controlled?

The identity contract also needs to define where the command should go.

A Jellyfin environment can contain multiple users, televisions, streaming devices, and active sessions. Selecting a destination by convenience alone can produce accidental success during testing and incorrect routing later.

One control mode uses an explicitly selected television:

Select TV Two
Play Movie Five
Pause Movie
Forward One Minute

After Select TV Two, subsequent commands remain associated with that destination until another television is selected.

This removes several runtime guesses. The system does not need to infer the intended room from microphone proximity, assume that the nearest active screen is correct, or switch targets simply because another client becomes active.

The selected destination becomes part of the command context.

When the Destination Becomes Part of the Command

Independent viewing areas require a stricter routing model.

Instead of keeping one shared selected target, the destination can be named directly in each command:

TV One Play Movie Five
TV Two Pause Movie
TV Three Open Movie List

Each zone is associated with its own configured Jellyfin user, device, and playlist, while the eligible live session is resolved at runtime from those identities.

Conceptually:

zone named in the command
    ↓
configured Jellyfin user + device
    ↓
eligible live session
    ↓
exactly one valid match?
    yes → continue
    no  → reject

The visible zone name is useful to the person speaking. Routing depends on the configured identities behind that name.

This also means that Movie Five can legitimately refer to different films in different zones, because each zone can have its own controlled playlist.

The command remains deterministic because both identities are explicit:

media identity       = Movie Five
destination identity = TV Two

Longer commands are a reasonable cost when independent rooms must remain independent.

Fail Closed When Identity Is Incomplete

Suppose only one television is active during development. A rule such as “use the first available session” may appear reliable.

The ambiguity becomes visible as soon as another client connects.

For device control, a guessed destination can be worse than a rejected command. I therefore prefer fail-closed behavior when a required identity is missing or ambiguous:

missing playlist slot     → reject
unavailable selected TV   → reject
ambiguous client session  → reject
incomplete zone mapping   → reject

This makes configuration problems visible instead of hiding them behind an unrelated session that happened to be available.

The Contract Behind Each Command

Once the identities are explicit, the command vocabulary can be described as a set of contracts rather than a list of phrases.

Command	Media identity	Destination identity	Failure rule
Play Movie Five	Playlist position 5	Selected television	Reject if the slot or target is unavailable
Pause Movie	Current media session	Selected television	Reject if no eligible session matches
TV Two Play Movie Five	Zone Two playlist position 5	Zone Two user and device	Reject if routing is missing or ambiguous
TV Three Open Movie List	Zone Three generated list	Zone Three device	Reject if the zone target is unavailable

The spoken phrase is the entry point. The contract defines the only meanings the phrase is allowed to have.

This way of thinking also helps when new commands are added. A command is incomplete until its identity source and failure behavior are defined.

Observable Success Belongs to the Contract Too

A command can be routed correctly and still depend on what the final Jellyfin client supports.

That matters because server acceptance and user-visible behavior are different layers of the system. The control model therefore treats the client as part of the execution path rather than assuming that a valid request guarantees a visible result.

Conceptually:

command
    + media identity
    + destination identity
    + eligible session
    + client capability
    = intended user-visible action

This does not require every Jellyfin client to expose identical capabilities. It requires the control layer to apply only actions that are supported for the target client.

The same contract idea still applies: support should be based on the behavior the user can actually rely on.

The Cost of Determinism

A deterministic interface gives up some flexibility.

The trade-offs are visible:

More flexible approach	Deterministic approach
Search the entire library by title	Use fixed playlist positions
Guess the intended television	Select or name the destination
Choose any available session	Require one exact eligible target
Hide incomplete configuration with fallbacks	Reject commands with unclear identity
Let the UI choose its own presentation order	Show the same order used by the command layer

The deterministic side asks the user to operate inside a smaller, more explicit model.

In return, the meaning of a valid command is easier to predict before it executes.

That is especially useful in home automation. The person speaking should not need to know which session the server currently considers active or why one client happened to win an internal selection rule.

What I Deliberately Left Outside the Contract

The current control model does not try to solve every voice-interface problem.

It does not attempt to provide:

• unrestricted full-library title search;
• conversational clarification between similar titles;
• automatic room detection;
• automatic destination selection;
• identical capabilities on every Jellyfin client;
• synchronized playback across several televisions.

Each of those features would require additional identity and state rules.

Full-library search would need ranking, pronunciation handling, confidence thresholds, and clarification. Automatic room detection would need a trustworthy relationship between microphones, users, rooms, and devices. Synchronized playback would introduce timing coordination and drift management.

Keeping them outside the current model makes the implemented behavior easier to explain and test.

Determinism Is a User-Facing Feature

Determinism may sound like an internal implementation detail. The user experiences it directly.

They experience it when:

• Movie Five refers to the same position shown on the television;
• Pause Movie reaches the selected target;
• TV Two Play Movie Five cannot affect TV One;
• an unavailable destination produces a clear failure instead of a command somewhere else;
• the interface shows the same state the command engine uses.

The model can be summarized as:

explicit media identity
+ explicit destination identity
+ allowed action
+ explicit failure behavior
+ observable target behavior
= predictable control

The broader lesson applies beyond Jellyfin.

When software controls real devices, every accepted command should have a precise identity contract. The interface becomes easier to trust when the person speaking and the system executing the request share the same definition of what should happen next.