Turning Spoken Commands into Safe Actions: Regex First, the Language Model Second

Introduction

I built a tiny voice‑controlled automation server at home to turn spoken phrases into actions on my personal web services. The key to keeping things safe is to filter the raw transcription with regular expressions before handing anything to a language model. This two‑step approach prevents the model from being asked to do anything unexpected, while still letting it handle the natural‑language nuances that regex alone can’t cover.

Architecture Overview

The pipeline looks like this:

1. Microphone → Speech‑to‑Text (STT) – I use the open‑source Vosk library for offline transcription.
2. Regex Guard – A whitelist of patterns that are allowed to proceed.
3. Prompt Construction – If the regex matches, I build a concise prompt for the LLM.
4. LLM Decision – The model returns a JSON‑encoded action description.
5. Executor – A tiny Flask endpoint validates the JSON schema and triggers the appropriate function.

All components run on a Raspberry Pi 4 in my garage, but the code works on any Linux box.

Speech‑to‑Text with Vosk

Vosk gives me low‑latency, offline transcription, which is important for privacy.

import vosk, sys, json, sounddevice as sd

model = vosk.Model("model")
rec = vosk.KaldiRecognizer(model, 16000)

def listen():
    def callback(indata, frames, time, status):
        if rec.AcceptWaveform(indata):
            result = json.loads(rec.Result())
            text = result.get("text", "")
            if text:
                process_command(text)
    with sd.InputStream(samplerate=16000, channels=1, callback=callback):
        sd.sleep(10_000)  # run for 10 seconds for demo

The process_command function is where the safety checks begin.

Regex Guard: The First Line of Defense

I keep a whitelist of allowed command patterns. Anything that doesn’t match is ignored and logged.

import re

# Simple patterns for demo – expand as needed
PATTERNS = {
    "turn_on": re.compile(r"turn on (the )?(?P<device>\w+)", re.I),
    "turn_off": re.compile(r"turn off (the )?(?P<device>\w+)", re.I),
    "set_temp": re.compile(r"set temperature to (?P<value>\d{1,2})", re.I),
}

def match_command(text: str):
    for name, pat in PATTERNS.items():
        m = pat.fullmatch(text.strip())
        if m:
            return name, m.groupdict()
    return None, None

Only exact matches pass through. This prevents the LLM from ever seeing ambiguous or malicious input.

Prompt Construction for the LLM

If a regex match succeeds, I hand a minimal prompt to the model. The prompt asks for a JSON payload describing the action. Keeping the prompt short reduces token usage and limits the model’s creative freedom.

import openai

openai.api_key = "YOUR_API_KEY"

SYSTEM_PROMPT = (
    "You are an assistant that converts a parsed voice command into a JSON action. "
    "Only output the JSON object, no extra text."
)

def ask_llm(intent, params):
    user_msg = f"Intent: {intent}, Params: {params}"
    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": SYSTEM_PROMPT},
                  {"role": "user", "content": user_msg}],
        temperature=0,
    )
    return response.choices[0].message.content.strip()

The LLM’s output looks like:

{"action": "turn_on", "target": "lamp"}

Executor: Validating and Running the Action

The Flask endpoint receives the JSON, validates it against a hard‑coded schema, and then calls the appropriate function.

from flask import Flask, request, jsonify
from jsonschema import validate, ValidationError

app = Flask(__name__)

SCHEMA = {
    "type": "object",
    "properties": {
        "action": {"type": "string", "enum": ["turn_on", "turn_off", "set_temp"]},
        "target": {"type": "string"},
        "value": {"type": "number"},
    },
    "required": ["action"],
    "additionalProperties": False,
}

def execute(action_obj):
    act = action_obj["action"]
    if act == "turn_on":
        turn_on_device(action_obj.get("target"))
    elif act == "turn_off":
        turn_off_device(action_obj.get("target"))
    elif act == "set_temp":
        set_thermostat(action_obj.get("value"))

@app.route("/run", methods=["POST"])
def run():
    try:
        payload = request.get_json(force=True)
        validate(instance=payload, schema=SCHEMA)
    except (ValidationError, TypeError) as e:
        return jsonify({"error": "invalid payload", "detail": str(e)}), 400
    execute(payload)
    return jsonify({"status": "ok"})

The turn_on_device, turn_off_device, and set_thermostat functions are thin wrappers around Home Assistant’s REST API – nothing exotic.

Putting It All Together

The process_command function ties the pieces:

def process_command(text: str):
    intent, params = match_command(text)
    if not intent:
        print(f"Rejected: {text}")
        return
    llm_output = ask_llm(intent, params)
    try:
        action_obj = json.loads(llm_output)
    except json.JSONDecodeError:
        print("LLM returned malformed JSON")
        return
    # Send to local Flask executor
    requests.post("http://localhost:5000/run", json=action_obj)

Because the regex guard runs first, the LLM never sees raw user speech. This dramatically reduces the attack surface: the model can’t be coaxed into generating harmful commands, and the executor only accepts a known schema.

Deployment Tips

• Run the Flask app behind a Unix socket and let Nginx proxy to it – this avoids exposing a raw port.
• Log rejected transcriptions to a file; they’re useful for expanding the whitelist.
• Version‑lock the Vosk model and OpenAI client to avoid accidental API changes.
• Use a virtual environment (python -m venv venv) and keep a requirements.txt – the stack is tiny:

  vosk
  sounddevice
  openai
  flask
  jsonschema
  requests

Lessons Learned

1. Regex first is not a silver bullet – it only works when the command space is bounded. For a personal home setup, that’s fine; for a broader product you’d need a more sophisticated intent classifier.
2. Keep LLM prompts minimal – the less you ask, the less chance the model will hallucinate extra fields.
3. Never trust the model’s output – always validate against a strict JSON schema before executing anything.
4. Offline STT matters – relying on a cloud service would re‑introduce privacy concerns and latency.

Conclusion

By front‑loading a regex whitelist, I turned a potentially risky voice‑to‑action flow into a deterministic, auditable pipeline. The language model still adds value by handling natural‑language variations, but it never gets a chance to interpret arbitrary speech. This pattern works well for my own home automations and can be adapted to small‑business kiosks or internal tools where the command set is known.

I build sites and automations for small businesses at dreamfuturestech.com.