DEV Community

develpmilk
develpmilk

Posted on

Kimi K3 Self-Hosting Cost Calculator: API vs 8 GPUs

#ai #api #webdev

Kimi K3 Self-Hosting Cost Calculator: API vs 8 GPUs

Kimi K3 can be self-hosted, but its public checkpoint is about 1.56 TB and the current vLLM recipe starts at eight high-end GPUs. For most teams, the useful comparison is not token price versus GPU rent. It is API cost per accepted task versus the complete cost of a reliable distributed deployment.

Why a calculator is more useful than a yes-or-no answer

Kimi K3 is an open-weight, 2.8-trillion-parameter Mixture-of-Experts model. It activates 104 billion parameters per token, selecting 16 of 896 routed experts, and supports a context window of up to 1,048,576 tokens.

Those numbers make self-hosting possible, but not simple.

The current vLLM recipe lists a starting topology of eight NVIDIA GB300 GPUs or eight AMD MI355X/MI350X-class GPUs. It recommends multi-node infrastructure for real production traffic. Moonshot's launch material recommends 64 or more accelerators for higher inference efficiency.

This leaves most teams with two realistic choices:

  1. Start with a hosted API and pay for measured usage.
  2. Operate a distributed GPU cluster and pay for capacity whether it is busy or idle.

The correct choice changes with traffic, cache reuse, output length, accepted-task rate, engineering cost, and utilization. The Python script below keeps those assumptions explicit.

What the cost model includes

The hosted side includes:

  • cache-hit input tokens;
  • cache-miss input tokens;
  • output tokens;
  • requests that are billed but fail application acceptance;
  • the number of requests per month.

The self-hosted side includes:

  • cluster cost per hour;
  • 730 provisioned hours per month;
  • platform and inference engineering;
  • networking and storage;
  • observability and security;
  • redundancy and idle headroom.

The output reports both total monthly cost and cost per accepted task.

Complete Python calculator

This example requires Python 3.11 or later and no third-party packages.

from dataclasses import dataclass


HOURS_PER_MONTH = 730
MILLION = 1_000_000


@dataclass(frozen=True)
class Workload:
    requests_per_month: int
    input_tokens_per_request: int
    output_tokens_per_request: int
    cache_hit_ratio: float
    accepted_task_ratio: float

    def validate(self) -> None:
        if self.requests_per_month <= 0:
            raise ValueError("requests_per_month must be positive")
        if self.input_tokens_per_request < 0:
            raise ValueError("input_tokens_per_request cannot be negative")
        if self.output_tokens_per_request < 0:
            raise ValueError("output_tokens_per_request cannot be negative")
        if not 0 <= self.cache_hit_ratio <= 1:
            raise ValueError("cache_hit_ratio must be between 0 and 1")
        if not 0 < self.accepted_task_ratio <= 1:
            raise ValueError("accepted_task_ratio must be above 0 and at most 1")


@dataclass(frozen=True)
class ApiPricing:
    name: str
    cache_hit_input_per_million: float
    cache_miss_input_per_million: float
    output_per_million: float


@dataclass(frozen=True)
class SelfHostedCost:
    cluster_hourly_rate: float
    engineering_per_month: float
    networking_and_storage_per_month: float
    observability_and_security_per_month: float
    redundancy_and_headroom_per_month: float

    def monthly_total(self) -> float:
        return (
            self.cluster_hourly_rate * HOURS_PER_MONTH
            + self.engineering_per_month
            + self.networking_and_storage_per_month
            + self.observability_and_security_per_month
            + self.redundancy_and_headroom_per_month
        )


def monthly_api_cost(workload: Workload, pricing: ApiPricing) -> float:
    workload.validate()

    total_input = (
        workload.requests_per_month * workload.input_tokens_per_request
    )
    total_output = (
        workload.requests_per_month * workload.output_tokens_per_request
    )

    cache_hit_input = total_input * workload.cache_hit_ratio
    cache_miss_input = total_input - cache_hit_input

    return (
        cache_hit_input / MILLION * pricing.cache_hit_input_per_million
        + cache_miss_input / MILLION * pricing.cache_miss_input_per_million
        + total_output / MILLION * pricing.output_per_million
    )


def accepted_tasks(workload: Workload) -> float:
    return workload.requests_per_month * workload.accepted_task_ratio


def print_api_report(workload: Workload, pricing: ApiPricing) -> float:
    total = monthly_api_cost(workload, pricing)
    accepted = accepted_tasks(workload)
    cost_per_accepted = total / accepted

    print(f"{pricing.name}:")
    print(f"  monthly API cost:       ${total:,.2f}")
    print(f"  accepted tasks:         {accepted:,.0f}")
    print(f"  cost / accepted task:   ${cost_per_accepted:,.4f}")
    return cost_per_accepted


def main() -> None:
    workload = Workload(
        requests_per_month=10_000,
        input_tokens_per_request=300_000,
        output_tokens_per_request=30_000,
        cache_hit_ratio=0.50,
        accepted_task_ratio=0.95,
    )

    moonshot = ApiPricing(
        name="Moonshot official API",
        cache_hit_input_per_million=0.30,
        cache_miss_input_per_million=3.00,
        output_per_million=15.00,
    )

    # The July 28, 2026 public CometAPI rate in the source package has
    # one input rate rather than separate hit/miss rates.
    cometapi = ApiPricing(
        name="CometAPI",
        cache_hit_input_per_million=2.40,
        cache_miss_input_per_million=2.40,
        output_per_million=12.00,
    )

    self_hosted = SelfHostedCost(
        cluster_hourly_rate=80.00,
        engineering_per_month=20_000.00,
        networking_and_storage_per_month=5_000.00,
        observability_and_security_per_month=3_000.00,
        redundancy_and_headroom_per_month=7_000.00,
    )

    print("Kimi K3 monthly cost model\n")
    moonshot_cost_per_task = print_api_report(workload, moonshot)
    print()
    cometapi_cost_per_task = print_api_report(workload, cometapi)
    print()

    self_hosted_total = self_hosted.monthly_total()
    print("Self-hosted scenario:")
    print(f"  monthly total:          ${self_hosted_total:,.2f}")
    print(
        "  break-even tasks vs Moonshot: "
        f"{self_hosted_total / moonshot_cost_per_task:,.0f}"
    )
    print(
        "  break-even tasks vs CometAPI: "
        f"{self_hosted_total / cometapi_cost_per_task:,.0f}"
    )


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it with:

python3 kimi_k3_tco.py
Enter fullscreen mode Exit fullscreen mode

For the supplied scenario, the script produces approximately:

Kimi K3 monthly cost model

Moonshot official API:
  monthly API cost:       $9,450.00
  accepted tasks:         9,500
  cost / accepted task:   $0.9947

CometAPI:
  monthly API cost:       $10,800.00
  accepted tasks:         9,500
  cost / accepted task:   $1.1368

Self-hosted scenario:
  monthly total:          $93,400.00
  break-even tasks vs Moonshot: 93,894
  break-even tasks vs CometAPI: 82,157
Enter fullscreen mode Exit fullscreen mode

Why the cheaper API changes with cache behavior

Moonshot's published Kimi K3 pricing in July 2026 is:

Usage Price per 1M tokens
Cache-hit input $0.30
Cache-miss input $3.00
Output $15.00

The source package records CometAPI at $2.40 per million input tokens and $12.00 per million output tokens on July 28, 2026.

CometAPI is below the official cache-miss input and output rates in this snapshot. It is not automatically cheaper when a workload has strong prefix-cache reuse, because Moonshot's official cache-hit input rate is much lower.

This is why the calculator accepts a cache-hit ratio. Change cache_hit_ratio from 0.50 to the value measured in your own application rather than treating one price row as the answer.

Before using the CometAPI price in a decision, check the live Kimi K3 model page. Pricing and availability are time-sensitive.

What the eight-GPU scenario still leaves out

An eight-GPU estimate is a planning floor, not proof of production readiness.

Kimi K3 routes tokens across experts, which creates all-to-all communication. The same nominal GPU count can produce different throughput depending on NVLink, RDMA, network topology, expert parallelism, prefill/decode separation, and concurrency.

A serious self-hosting test should also include:

  • model download and startup time;
  • storage replication and recovery;
  • KV-cache memory for the context lengths actually used;
  • deployment rollback and weight upgrades;
  • node failure and degraded-cluster behavior;
  • tool-call validation and retries;
  • capacity reserved for traffic bursts;
  • engineering on-call coverage.

Do not use the theoretical maximum context by default. Long prompts increase prefill work, KV-cache demand, latency, and the amount of capacity unavailable to concurrent requests.

When self-hosting becomes credible

Self-hosting is worth a load test when at least one of these conditions is true:

  1. Demand is sustained and predictable enough to keep the cluster productively busy.
  2. The organization already operates distributed MoE inference and high-bandwidth GPU networking.
  3. Data-path control or a dedicated environment is a hard requirement.
  4. The team needs direct control over model versions, runtime configuration, or fine-tuned weights.
  5. Measured hosted spend approaches the complete cost of an equivalent reliable internal service.

For new or variable traffic, an API is usually the safer baseline. It gives the team real token, cache, latency, retry, and acceptance data before a hardware commitment is made.

Key takeaways

  • Kimi K3's open weights create deployment freedom, not a low-cost local setup.
  • The current serving floor starts at eight high-end GPUs; efficient production may require a much larger topology.
  • Prefix-cache reuse can change which hosted route is less expensive.
  • Compare cost per accepted task, not raw token price or GPU rent.
  • Buy infrastructure only after a production-shaped load test supports the complete TCO case.

Top comments (0)