Orquesta CLI: Local LLM Management With Cloud Sync

Originally published at orquesta.live/blog/orquesta-cli-local-llm-management-cloud-sync

Managing large language models (LLMs) locally can be a challenging task, especially when you need to keep configurations in sync across an organization. This is where Orquesta CLI steps in, offering a streamlined solution for local LLM management with seamless cloud dashboard synchronization.

Why Local LLM Management Matters

Running LLMs locally has significant advantages. It ensures that your sensitive code and data remain within your infrastructure, providing a higher level of security and compliance. Orquesta CLI enables you to run powerful LLMs like Claude, OpenAI, Ollama, and vLLM right on your local machine. This means you get the full power of these models without the latency and privacy concerns that come with cloud-based execution.

Key Features of Orquesta CLI

Orquesta CLI is designed to make local LLM management straightforward and efficient:

• Local Execution: Run any supported LLM locally, ensuring your data never leaves your infrastructure.
• Dashboard Sync: Automatically sync configurations and logs with the Orquesta cloud dashboard for a unified view and management.
• Prompt History Tracking: Keep a comprehensive log of all prompts and interactions with your LLMs.
• Org-Scoped Tokens: Use organization-scoped tokens to manage access and permissions efficiently.
• Bidirectional Config Sync: Automatically update local configurations from the cloud and vice versa, ensuring consistency across deployments.

Setting Up Orquesta CLI

Getting started with Orquesta CLI is simple. First, ensure that your environment meets the prerequisites for running the LLMs you plan to use. Once you have the necessary dependencies, you can install Orquesta CLI with a single command:

pip install orquesta-cli

With Orquesta CLI installed, you can configure your local environment to sync with the cloud dashboard. This involves setting up org-scoped tokens, which provide a secure way to authenticate and manage access to your LLMs.

Usage Example: Running Claude Locally

Let's walk through a typical workflow where we configure and run the Claude LLM using Orquesta CLI. First, initialize your configuration:

orquesta init

This command sets up the necessary configuration files and prompts you to enter your organization’s token. Once configured, you can run Claude locally by specifying the model and any additional parameters:

orquesta run --model cl-claude --input "What is the capital of France?"

The output is instantaneous, with all data processed locally. Any updates to your configuration or prompts are automatically synced with the Orquesta dashboard, where team members can review and manage interactions.

Syncing Configs to the Cloud

Orquesta CLI’s ability to sync local configurations with the cloud dashboard is a game-changer for collaborative teams. Every time you update your LLM settings or add new models, these changes are reflected in the cloud, ensuring that everyone on your team has access to the most up-to-date configurations.

How Bidirectional Sync Works

The bidirectional sync feature of Orquesta CLI ensures that any changes made in the cloud dashboard are also pulled down to your local environment. This eliminates the risk of configuration drift, which can occur when team members work in silos or forget to update local settings.

Here’s how it works:

1. Config Push: When you make changes locally, Orquesta CLI pushes these updates to the cloud dashboard.
2. Config Pull: At regular intervals or on command, your local setup pulls the latest configurations from the cloud.

This seamless synchronization allows for a unified, consistent setup across all team members, reducing errors and improving collaborative efficiency.

Tracking Prompt History

One of the standout features of Orquesta CLI is its ability to track prompt history. Every prompt and its corresponding response are logged, providing a valuable audit trail that can be reviewed and analyzed. This is particularly useful for debugging, compliance, and understanding how models are used within your organization.

The Importance of Prompt History

Tracking prompt history offers several benefits:

• Auditability: Easily review past interactions for compliance and troubleshooting.
• Insight Generation: Analyze prompt patterns to optimize model usage and performance.
• Collaboration: Share insights with team members to improve collective understanding of LLM capabilities and limitations.

Conclusion

Orquesta CLI transforms the way teams manage and deploy LLMs by offering local execution with cloud-sync capabilities. It’s a robust tool for organizations that need the power of LLMs without compromising on security or flexibility. With features like bidirectional config sync and prompt history tracking, Orquesta CLI ensures that your team can collaborate effectively while maintaining control over their local setup.

Whether you're a solo developer or part of a large team, Orquesta CLI provides the tools you need to harness the full potential of LLMs locally, with the added benefits of cloud-based management and collaboration.