Kimi K3 is self-hostable, but I would treat it as a distributed-inference project, not a routine model deployment. The public checkpoint is approximately 1.56 TB, the official vLLM baseline starts at eight NVIDIA GB300 or eight AMD MI355X/MI350X GPUs, and Moonshot recommends 64 or more accelerators in a supernode configuration for efficient production inference. My default would be hosted access until workload measurements justify owning that infrastructure.
Start With the Workload, Not the Weights
Moonshot introduced K3 in its technical blog on July 16, 2026, and released the full weights by July 27. That makes deployment under your own account possible. It does not establish that doing so is economical. With self-hosting, your team owns capacity, serving, scaling, security, upgrades, monitoring, and recovery. With an API, the provider handles the underlying cluster, while you retain responsibility for application behavior and the provider relationship.
I would compare the two routes on cost per accepted equivalent task. Raw token prices ignore retries and quality failures; GPU rent ignores engineering and idle capacity. Even endpoints using identical weights are not economically equivalent when one misses latency or reliability requirements.
What You Actually Have to Serve
K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 104 billion activated parameters per token, 896 routed experts, native multimodal capabilities, MXFP4 weights, MXFP8 activations, and a context window of up to 1,048,576 tokens. It activates 16 routed experts per token. The 104B figure describes active computation, not the model’s storage requirement: routing can select different experts during generation, so the complete expert set remains part of the deployment.
The official repository publishes the weights under the custom Kimi K3 License. Public serving paths include vLLM, SGLang, and TokenSpeed. Each still needs validation against the intended workload; engine support is not a throughput guarantee.
Hardware, Storage, and the Interconnect
The official vLLM recipe requires vLLM 0.27.0 or later, the Kimi K3 image, and its documented deployment profiles. Its minimum is 8x GB300 GPUs on NVIDIA or 8x MI355X/MI350X on AMD ROCm, with multi-node deployment recommended for production traffic. This is a documented serving floor, not a promise that eight GPUs will meet your concurrency, context-length, latency, and redundancy targets. It is not official guidance for a single-GPU deployment.
Moonshot’s architecture guidance recommends supernodes with 64 or more accelerators for higher inference efficiency. Expert parallelism creates substantial all-to-all traffic, so aggregate GPU memory is only part of the requirement. The vLLM recipe identifies deepep_v2 for RDMA environments and flashinfer_nvlink_one_sided for NVLink-based cross-node communication. Eight GPUs on a slow network are not operationally interchangeable with eight GPUs in a tightly connected system.
The Hugging Face repository is approximately 1.56 TB. The theoretical four-bit parameter-storage floor is:
2.8 trillion parameters × 4 bits ÷ 8
= 1.4 trillion bytes
= about 1.4 TB, or 1.27 TiB
Neither figure is a complete runtime-memory budget. Serving also needs quantization metadata, activations, KV cache, communication buffers, and concurrency headroom. I would benchmark tensor parallelism, expert parallelism, topology-specific MoE backends, and the documented disaggregated prefill/decode profiles before treating any proposed cluster as production capacity.
The Application Contract Matters on Both Routes
A hosted endpoint removes most cluster operations, but it does not remove schema validation, retries, state management, or latency measurement. The vLLM recipe notes that K3 can occasionally generate a tool-call format its own parser does not expect, and recommends validation and retry handling. Those failures belong in the cost model, not just an error dashboard.
According to the API quickstart, K3 always reasons. Supported reasoning_effort values are low, high, and max, with max as the default. Reasoning can increase generated-token volume, but there is no fixed overhead multiplier worth assuming. Measure reasoning and output tokens on your evaluation set. For multi-turn conversations and tool calls, preserve the complete assistant message, including reasoning_content and tool_calls, rather than replaying only the visible answer.
The million-token context is not free capacity. Moonshot does not impose a higher per-token tier solely for longer context, but longer prompts still increase input usage, prefill work, KV-cache demand, latency, and concurrency pressure. For self-hosting, I would explicitly tune max-model-len, concurrency, and memory utilization around real requests instead of enabling the maximum by default.
Price the Hosted Baseline With Cache Behavior Included
As of July 2026, Moonshot’s published prices are $0.30 per 1M cache-hit input tokens, $3.00 per 1M cache-miss input tokens, and $15.00 per 1M output tokens. I would calculate request cost directly from billed categories:
API cost =
(cache-hit input tokens / 1,000,000 * $0.30)
+ (cache-miss input tokens / 1,000,000 * $3.00)
+ (output tokens / 1,000,000 * $15.00)
A request with 300,000 input tokens and 30,000 output tokens costs $1.35 with entirely cache-miss input, or $0.54 with entirely cache-hit input. Real requests usually fall between those cases. Reusing an unchanged prefix matters, as does the provider’s caching implementation; a headline input rate alone is a poor comparison.
For a unified multi-model API, CometAPI’s K3 listing quotes July 2026 rates of $2.40 per 1M input tokens and $12.00 per 1M output tokens, 20% below Moonshot’s standard cache-miss rates. That is not a universal 20% saving: Moonshot’s $0.30 cache-hit input rate can make the official route cheaper for cache-heavy workloads. Check live pricing and compare actual bills for identical requests, including reasoning output, retries, cache hits, and accepted-task rate.
Build a Complete Self-Hosting Budget
There is no universal K3 hosting price. Cluster size, contract terms, productive utilization, network topology, storage, engineering, and reliability targets all change the answer. My monthly model would add accelerator or cluster charges, platform engineering, inference operations, networking and storage, observability and security, and redundancy and idle headroom.
For a minimum-size eight-GPU planning scenario, the following assumes 730 hours per month and hypothetical cluster rates. These are not quotations, measured throughput figures, or budgets for Moonshot’s recommended 64+ accelerator production topology. The request counts are rounded spending equivalents using the example above.
| Assumed Cluster Rate | Monthly Infrastructure | Requests at $1.35 Each | Requests at $0.54 Each |
| --- | --- | --- | --- |
| $80/hour | $58,400 | 43,300 | 108,100 |
| $120/hour | $87,600 | 64,900 | 162,200 |
| $160/hour | $116,800 | 86,500 | 216,300 |
Even this infrastructure-only scenario exceeds $58,000 per month at its lowest assumed rate. Engineering, monitoring, networking, redundancy, and idle capacity push the threshold higher. A larger production topology needs a separate budget and its own measured capacity.
I would track productive utilization = cluster-hours spent on accepted workload / total provisioned cluster-hours. There is no universal GPU-utilization percentage that makes self-hosting cheaper. High utilization does not help if requests miss quality or latency targets; lower utilization can still be acceptable when hardware is already committed for other workloads. Hardware ownership, measured throughput, redundancy, and service targets all matter.
The useful spending comparison is break-even accepted tasks = total monthly self-hosted cost / hosted API cost per accepted equivalent task. Include failures, retries, latency violations, degraded outputs, and human review on both sides. This comparison only becomes meaningful after confirming that the proposed cluster can deliver the required work at equivalent quality and reliability.
Review the License Before Committing Infrastructure
I would call K3 open-weight, not fully open source. Its custom license grants broad rights to use, copy, modify, fine-tune, deploy, distribute, sublicense, and sell the software and weights, subject to its conditions and applicable law. The business-specific conditions deserve a separate review.
“Model as a Service” means third-party access to inference or fine-tuning that gives meaningful control over inputs, parameters, or training data. Embedded product features and mere relaying to models hosted by others are excluded. For a licensee and affiliates operating a MaaS business, more than $20 million in aggregate revenue over any consecutive 12 months triggers a separate Moonshot agreement requirement before commercial use of the software or derivatives.
Visible attribution is a different condition: a commercial product or service with more than 100 million monthly active users or more than $20 million in monthly revenue must prominently display “Kimi K3.” Internal use and access through Moonshot’s official products or certified inference partners are exempt from Sections 2 and 3. Ordinary commercial applications and internal deployments are not prohibited by default, but direct-access businesses and teams approaching those thresholds should have counsel review the product design and corporate structure. Hosted access also requires reviewing the provider’s terms, architecture, logging, and retention policies.
My Deployment Gate
I would build 30 to 50 representative tasks covering the actual coding, long-context, vision, and tool-calling mix, then collect hosted measurements for at least one week. Record input, cache-hit, and output tokens alongside latency, retries, errors, and acceptance rate. Keep prompts, tool schemas, and acceptance criteria unchanged across routes.
Next, load-test the proposed self-hosted topology using the intended context limits, concurrency, parallelism, and reliability settings. A single-user demonstration is not enough. Calculate the full monthly cost and test node loss, deployment rollback, queue growth, long-context bursts, and malformed tool calls. Include engineering time, idle headroom, and failure recovery over a representative operating period.
I would stay with an API for new or variable traffic, uncertain break-even economics, limited distributed-MoE experience, or a need to deploy without GPU procurement, provided the provider’s data and service terms fit. Self-hosting becomes credible with sustained demand, existing distributed infrastructure and inference engineers, measured cost advantages, or hard requirements for data-path, retention, version, scheduling, runtime, or fine-tuned-weight control. Legal approval is part of that decision. Strategic control can justify a higher price, but I would record that trade-off explicitly.
A hybrid can serve predictable baseline demand internally and use an API for bursts, maintenance, or regional failures. I would only add it when the capacity or resilience benefit is measured and prompts, tools, acceptance tests, and behavior remain portable. Routing and observability add real work. The deployment decision should follow that evidence, not the fact that the weights are downloadable.
Originally published at cometapi.com
Top comments (0)