DEV Community

Cover image for When to use an open model vs a proprietary LLM
Conor Bronsdon
Conor Bronsdon

Posted on Originally published at chainofthought.show

When to use an open model vs a proprietary LLM

Pick the model from the constraint the task cannot drop. Open weights fit when you need control over training, control over how your data is used, a zero data retention policy, on-prem hardware, or a specific size and speed. Proprietary models fit when vision quality, time to a first working call, or quality on that task matters more, and Aishwarya Srinivasan marks DeepSeek V3 as the point where the performance and price ratio of that open, customized path went up.

Aishwarya Srinivasan was then Head of AI Developer Relations at Fireworks AI. That conversation aired in October 2025 on Chain of Thought episode 42. Angie Jones was then VP of Engineering for AI Tools and Enablement at Block. That conversation aired in January 2026 on episode 48.

How do you decide open weights or a proprietary API?

Name the constraint first, then the model. Srinivasan said that after DeepSeek V3, the performance and price ratio of building in a cost efficient, customized way, with full control on an open source model, went up compared with a proprietary model. R1, then V3.1, followed. The jobs where she sees people switch:

  • Control over how the model is trained.
  • Control over how data pushed into the system is used.
  • A zero data retention policy.
  • Running on prem.
  • Customizing size and speed.

Teams still keep proprietary models. Developer experience, and how easy it is to get up and running, still pulls people toward closed APIs. On vision, proprietary models give really good performance, and vision is harder for the open models out there. Public benchmarks still show some performance gap. Enterprises and startups she watches usually do not run open models out of the box. They use supervised fine tuning, reinforcement learning fine tuning, or synthetic data when they lack a dataset.

Policies differ by provider, but in most cases, as she understands it, a proprietary model does not come with a zero data retention policy.

after the DeepSeek v3 model came, we did see that inflection point where the performance and price ratio of what you can really build at a very cost efficient and a very customized manner, something that you have full control on with an open source model compared to a proprietary model, it just went up.

Aishwarya Srinivasan, then Head of AI Developer Relations at Fireworks AI, on Chain of Thought ep 42

Architecting AI Agents: The Sh… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · October 8, 2025 · 53m

favicon podcasts.apple.com

How do you route several models inside one company?

Keep a hosting layer and a default per job, and leave room to change the default. Strengths differ. Anthropic's Claude models are fantastic at software development, so Block's engineers typically stick there. Job functions that rely more on reasoning might want a GPT model. Databricks provides their model hosting service, with Gemini models from Google, GPT models from OpenAI, and the Anthropic models.

Local open source models are a fan favorite in some niche circles, and they run on the person's machine. Frontier models in the cloud use the provider's resources. For agentic software using some local open models, Jones said larger models were needed, prompting some engineers to upgrade their machines. She called that group a very small percentage.

Goose, Block's open source agent, is model agnostic because Block is not selling the tokens, so they do not have a dog in the fight. Inside one workflow, a person can switch between GPT and Claude.

You know, Conor, no one has won this race yet. Um, in any given week, if you say what's the best model out, it'll be from a different provider, right? And so it would be foolish of us to kind of lock in on one specific one.

Angie Jones, then VP of Engineering for AI Tools and Enablement at Block, on Chain of Thought ep 48

What check stops a benchmark from picking the model for you?

Write down the user step and the failure you will measure before you bind a model id. Srinivasan said you cannot look at a model benchmark, decide it is good enough, and tie models together. She can break any model with quirky, twisted prompts. Most of the use cases she sees do not run on one particular model.

At a Fireworks Dev Day, Notion's head of AI engineering said parts of Notion AI needed latency and response quality they could not get from proprietary models. They fine tuned the models on the Fireworks AI platform and ran them at very low latency and still used a combination of models.

A minimal sketch to adapt, not a drop-in library.

def route_model(task):
    """Hypothetical helper. Labels are local names, not a vendor API."""
    if (task.control_training or task.control_data_use or task.zero_retention
            or task.on_prem or task.tune_size_or_speed):
        return "open_weights"
    if task.needs_vision or task.needs_fast_setup:
        return "proprietary_api"
    if task.job == "software":
        return "software_default"  # Block engineers: Anthropic models
    if task.job == "reasoning":
        return "reasoning_default"  # some roles may want GPT
    return "unset"
Enter fullscreen mode Exit fullscreen mode

FAQ

Should we standardize on one model provider?
Jones said no one has won this race yet, and that it would be foolish to lock in on one specific provider. In any given week the best model can come from a different provider, and different jobs want different strengths.

What changed with DeepSeek V3?
Srinivasan called V3 an inflection point in the performance and price ratio for cost-efficient, customized, fully controlled open models versus proprietary ones. R1 and V3.1 followed.

Are local open models the right default for agent work?
Some people use them, depending on their hardware. Jones said agentic software needs the larger local models, and only a very small percentage of engineers had upgraded their machines to run them. Cloud frontier models use the provider's machines.

Takeaway

  • Write the binding constraint on the task before you choose a model family.
  • Use open weights when you must control training, data use, retention, hardware, size, or speed.
  • Use a proprietary model when vision, setup time, or quality on that task is the binding constraint.
  • Keep more than one provider behind the hosting layer, with a default a person can change.
  • Revisit the route when a new open model moves the price and quality line.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Top comments (0)