Article also available over here as well.
For the longest time I've used iTerm2 as a replacement for Terminal. It's fast, it's native, it's not yet another lipstick-on-an-Electron-wrapper type of thing. Only Ghostty comes close to it, and even though it's faster and resizes better, it misses some of the features I've grown to depend on.
One of those is the new AI Chat feature.
From the documentation:
The assistant can interact with the terminal, subject to your permission. It can also explain command output, adding annotations right in the terminal.
Sounds cool, right?
In case you're wondering why anyone would even want such a thing, I want you to imagine a world in which you don't need to remember, memorize, google, etc any ffmpeg command ever again. That's the world I live in and it's glorious.
One thing I am aware of though is the potential privacy risks of allowing a random LLM access to my console history. So I looked into hooking it up with my dedicated Azure OpenAI instance for its added privacy. Having the ability to set up a series of prompt injection shields is nice as well.
Here's how I did it.
Deploying Azure OpenAI
I'm assuming you have some sort of an Azure AI resource, be it Azure AI Foundry or Azure OpenAI, both work so don't worry too much about picking the right one (but if you're creating something new, go for AI Foundry 😉).
Deploy your favorite OpenAI small-ish model ever which in my case it's gpt-4o-mini. Make sure to pick some reasonable limits for the Tokens per Minute Rate Limit.
My recommendation is to also set up a custom Content filter, to take advantage of the prompt injection shielding (just in case 🤞). You can do this from the Guardrails + Controls blade, Content filters tab, Create content filter, pick anything other than the default name (cli-shield has a nice ring to it), set the violence/hate/sexual/self-harm settings to whichever snu-snu preferences you may have but set both Prompt shields for jailbreak attacks and Prompt shields for indirect attacks to Annotate and block. Don't bother activating Spotlighting, since that's only available for the Chat Completions api and we'll be using the Responses one.
What this filtering achieves is that if someone or something manages to inject some nasty LLM instructions in your command line history or context, you'll be spared from the worst effects of it.
One more thing to note is that setting overly aggressive filters might cause requests like kill process x to fail, so be ready to tweak these settings if needed.
Set whatever options you prefer for the Output filter, but you'll probably want to set the Streaming mode to Asynchronous filter to avoid having to wait for the filter to run before receiving the response tokens.
Hit Next, apply the filter to your model, hit Create filter and you're good to go.
Make a note of your model's Deployment Name plus your resource's Name and API key 1, which are available in the Home blade.
Configuring iTerm2 to use said Azure OpenAI
Settings -> General -> AI is where we're going.
First, make sure your AI Plugin is installed and working ✅. Check Enable generative Al features.
Set API Key... to the resource's API key from above. Uncheck Always use the recommended model from: and choose to Configure Al Model Manually.... This is the tricky part.
You'll need to fill in the Model to the name of the model you've deployed before. Leave the Token Limit as-is (assuming you've set a standard deployment name like gpt-4o-mini instead of something inventive and unique like rainbow-colored-bugaboo in which case, ymmv). The URL should be https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/responses and the API used should be Responses.
I've only allowed the Function Calling and Streaming Responses features, and disabled everything else, including No File Uploading or Vector Store (they just didn't make sense for a console helper).
One other thing I've done and you might want to do as well is explicitly disable some of the features. It's never fun to mistakenly approve some action you never should have approved. Long story short, here's how this looks for me
| Capability | Setting | Notes |
|---|---|---|
| Check Terminal State | Ask Each Time Run | |
| Commands | Ask Each Time | |
| Type for You | Never Allow | Running commands is enough until I get a better feel of how this feature works |
| View History | Ask Each Time | |
| View Manpages | Ask Each Time | |
| Write to Clipboard | Never Allow | My clipboard is my own thx |
| Write to Filesystem | Never Allow | My files too |
| Act in Web Browser | Never Allow | I don't use iTerm2's built-in browser so this isn't needed |
Conclusion
Setting this up simplified some of my workflows, since I never really bothered learning the obscure parts of the Bash syntax (by obscure I mean whatever I haven't absorbed already 🤫). Instead of copy-pasting questions to LibreChat, now I can focus on actually solving the issues.
The iTerm2 configuration UI is still a bit clunky, too many clicks to set this up and it doesn't offer a simple way to configure & switch between multiple models. But it's still nicer than the alternative.
You should try it out.
Top comments (0)