DEV Community: Jake Tao

Reducing an $823 AWS Bill in Two Days

Jake Tao — Fri, 14 Aug 2026 18:25:02 +0000

This article was originally published on my blog. For the latest version and future updates, please visit the original post.

Cloud cost optimization isn’t simply a matter of “switching to smaller servers.” It starts with understanding the actual workload, then using billing data and monitoring to identify the true sources of cost, and finally verifying through rolling deployments, health checks, and cold-start tests that the optimized system remains reliable.

Background: Fixed Costs in a Production Environment

This is a SaaS system that has gone live and supports real business operations. It maintains both Production and Staging AWS environments and includes a Dashboard, Gateway API, OpenAPI, SuperAdmin, asynchronous Workers, and infrastructure such as Aurora, Valkey, ALB, NAT Gateway, and CloudWatch.

Although the system’s overall load and resource utilization are relatively low, the UnblendedCost recorded by AWS Cost Explorer for the full billing cycle before optimization reached $822.96.

The goal was to bring monthly infrastructure costs down to roughly $100–$200 while preserving the ability to scale quickly as the business grows.

These two goals naturally conflict. A fully provisioned high-availability architecture creates significant fixed costs regardless of actual traffic. On the other hand, consolidating every service onto a single inexpensive server might reduce the bill further, but would introduce a single point of failure and increase future migration costs.

We therefore established three constraints before making any changes:

Production changes must not cause prolonged downtime;
The path to future scaling must remain intact;
No resource should be removed based solely on intuition.

Look at the Bill First, Not the Servers

We began by breaking down the pre-optimization billing cycle by AWS service:

Amazon ECS: $276.59
RDS: $174.79
EC2 - Other: $142.32
Elastic Load Balancing: $77.82
CloudWatch: $57.10
VPC: $41.84

Further analysis revealed that within the EC2 - Other category, hourly fees and data processing charges for just two NAT Gateways accounted for $135.72.

ALB LCU traffic charges were minimal, with most of the cost coming from the fixed hourly fees of running multiple load balancers. CloudWatch costs, meanwhile, were almost entirely attributable to metric monitoring.

This showed that the high bill was not caused by a large volume of requests. Instead, it came from simultaneously paying for multiple always-on compute resources, ingress and egress infrastructure, database instances, and enhanced monitoring.

The optimization priority therefore became clear: address hourly fixed costs first, then optimize smaller usage-based charges.

First, Determine Which Capabilities Are Non-Negotiable

One of the easiest mistakes in cloud cost optimization is to rank resources purely by price without first defining the system’s operational boundaries.

Production already supports real business activity, so the database, cache, object storage, secrets management, and a stable public ingress layer all needed to remain. What could be adjusted were replica counts, task sizes, and redundant infrastructure.

Staging serves a different purpose. It exists for testing and release validation and does not need to be available 24/7. That means startup latency can be traded for substantially lower always-on costs.

We also chose not to migrate every service onto a single low-cost instance. Such an approach could reduce the bill further, but it would concentrate the application, ingress, and deployment process into a single failure domain and disrupt the existing deployment workflow.

By keeping ECS, ALB, and Aurora, future scaling can still be handled by increasing task counts, task sizes, or database capacity without redesigning the entire architecture.

Every optimization therefore had to answer three questions:

Does the current workload actually require this capacity?
What capability do we lose if we remove it?
Can we restore that capability quickly when the business grows?

Those questions matter more than asking whether another ten dollars can be removed from the bill.

Consolidate ALBs While Keeping a Mature Ingress Layer

An Application Load Balancer can be thought of as the traffic coordinator at the front of the system.

Initially, both Production and Staging had multiple ALBs serving different subsystems. At low traffic levels, the fixed hourly cost of each ALB was more significant than the actual traffic-related charges.

We ultimately retained only one shared ALB per environment. Using Host Header and Path Rules, requests for the Dashboard, Gateway, OpenAPI, and SuperAdmin are forwarded to their respective Target Groups.

This allows multiple domains and independent services to continue operating separately while sharing the same ingress layer.

During the migration, we first created the new forwarding rules and Target Groups and confirmed that the targets were healthy. Only then did we update DNS.

After verifying all domains, certificates, and application entry points, we removed the old ALBs and their associated resources.

We also evaluated Cloudflare Tunnel, but it would have introduced new account permissions, ingress dependencies, and troubleshooting workflows. For this system, retaining AWS-native ALB infrastructure was the more predictable choice.

The most important failure mode during this type of migration is a 503. Validation therefore cannot stop at confirming that the ALB itself is marked as Active.

We also verified that:

Listener Rules routed to the correct Target Groups;
Health check paths returned successful responses;
DNS had propagated to the new ingress layer.

The old ALBs remained available until migration validation was complete, preserving a clear rollback path if anything went wrong.

Use Monitoring Data to Guide Fargate Downsizing

The Production API and Web services were originally configured conservatively.

Monitoring data from the 24 hours before downsizing showed:

API CPU averaged approximately 2.60%, with a peak of 5.86%;
Web CPU averaged approximately 2.36%, with a peak of 5.60%;
Peak memory utilization remained below 4%.

We gradually reduced both the API and Web services to 0.25 vCPU / 0.5 GB. OpenAPI and SuperAdmin were already running at the minimum task size and were left unchanged.

Each change used an ECS rolling deployment: new tasks were started first, ALB health checks were allowed to pass, and only then were the old tasks drained.

Monitoring during the 24 hours after downsizing showed:

API CPU averaged approximately 4.84%, with a peak of 12.88%;
Web CPU averaged approximately 5.34%, with a peak of 17.28%;
Peak memory utilization for both remained below 7%.

Production ultimately retained four online tasks at the minimum specification, with estimated monthly Fargate costs of approximately $36.

Idle Workers were not deleted. Instead, they were configured with desired=0, allowing them to be restored quickly when needed.

Reducing the task specifications did not change the container images, network interfaces, or deployment model.

If traffic increases in the future, the first scaling option is simply to increase the desired count. If individual requests begin requiring substantially more CPU or memory, a larger Task Definition can then be deployed.

Keeping that scaling path intact was one of the prerequisites for reducing always-on capacity.

There is, however, a clear trade-off. A single task does not provide high availability in the traditional multi-replica sense. ECS will automatically replace a failed task, but the service may experience a brief interruption during recovery.

As traffic, SLA requirements, or business criticality increase, restoring multiple replicas for critical services should be one of the first infrastructure changes.

The Reader Is Valuable, but Is It Worth Keeping Always On?

The Production Aurora cluster originally had one Writer and one Reader.

The Writer handles writes and transactions and can also serve reads. The Reader uses the same Aurora distributed cluster storage but runs as a separate database instance for read-only queries.

It can reduce read pressure on the Writer and can also be promoted if the Writer fails, reducing recovery time.

After examining application connections, database sessions, and actual request paths, we confirmed that the Reader was receiving real production reads. It was not an idle resource.

The actual question was:

Does the current read volume and failover benefit justify the cost of keeping another database instance running continuously?

Based on the overall workload and connection volume, the Reader was offloading only a limited amount of traffic, and the Writer still had sufficient capacity to absorb those reads.

Before deleting the Reader, we safely redirected read connections back to the Writer, performed a rolling application restart, and validated the read path.

We then queried PostgreSQL’s pg_stat_activity and confirmed that only the internal rdsadmin connection remained on the Reader. Once all business traffic had drained, the Reader was removed.

Production now retains a single Serverless v2 Writer with a minimum capacity of 0.5 ACU.

Based on the current pricing model, removing the Reader is expected to save approximately $43.20–$44.64 per month.

The trade-off is losing a database instance that can immediately take over during a Writer failure. Aurora storage remains distributed across Availability Zones, but recovery from a Writer failure may take longer without an existing Reader available for promotion.

If read pressure or recovery requirements increase in the future, a Reader can be recreated and read/write separation restored.

This database change also required careful Terraform state management.

The original configuration used count to manage the two database instances. Simply changing the value from 2 to 1 could cause Terraform to interpret array index changes as a request to replace the wrong instance.

We therefore migrated to stable for_each identifiers and used a moved block to preserve the Writer’s resource identity.

Only after confirming that the Terraform Plan would delete the intended Reader—and nothing else—did we apply the change.

Make Staging Truly On-Demand

Staging was originally configured to start and stop automatically on weekdays.

However, the test environment is not used every day, so scheduled startups still generated unnecessary compute costs.

We ultimately disabled automatic startup, set all four Staging ECS services to a default desired count of 0, and configured Aurora with a minimum capacity of 0 ACU, automatically pausing after 15 minutes of inactivity.

We also created a unified script supporting:

start
status
stop

This allows the entire Staging environment to be managed through a single command.

After completing the script, we ran a full recovery drill covering startup, health checks, endpoint validation, and shutdown. All four public application endpoints passed validation.

An on-demand environment is only reliable if it has been proven capable of starting successfully from zero.

For routine testing, we now run:

staging-on-demand.sh start

After the services and database have resumed, testing can begin. When testing is complete:

staging-on-demand.sh stop

This is more reliable than manually modifying individual services in the AWS Console and makes it less likely that someone will forget to shut Staging down after testing.

Before Removing NAT, Prove the System Does Not Depend on It

A NAT Gateway typically provides private subnets with outbound access to the public internet. AWS charges both an hourly fee and a data processing fee for traffic passing through it.

During the billing cycle before optimization, the two NAT Gateways generated approximately $77.38 in hourly charges and $58.34 in data processing fees.

We migrated ECS Fargate tasks to public subnets and assigned them public IP addresses, while keeping inbound access tightly restricted.

The task security groups do not allow inbound connections from public CIDR ranges. Application ports can only be accessed from the ALB Security Group.

The database and Valkey remain in private networking.

We also added free S3 Gateway Endpoints to the private route tables in both VPCs, reducing private-network dependency on NAT.

Running a task in a public subnet does not mean its application ports are directly exposed to the internet.

The public IP provides an outbound path for tasks to reach services such as ECR and Secrets Manager. Inbound access is still governed by the security group.

During final verification, task security groups contained no 0.0.0.0/0 or public IPv6 inbound rules, and application ports were reachable only from the ALB Security Group.

Before deleting the NAT Gateways, we verified:

Container image pulls;
Secrets access;
CloudWatch Logs;
Database connectivity;
Valkey connectivity;
ALB health checks;
Production application domains.

Staging successfully performed a cold start from zero without NAT, and Production successfully rolled out a complete new set of tasks.

Only after confirming that the application no longer depended on NAT did we remove both NAT Gateways and their dedicated Elastic IP addresses.

After accounting for the additional public IPv4 charges for Fargate tasks, the expected reduction in fixed costs is approximately $60–$70 per month.

If traffic remains similar to the previous billing cycle, the associated NAT data processing charges should also largely disappear.

This design is appropriate for the system’s current security requirements. If future requirements include fixed egress IP addresses, strict private-network compliance, or more sophisticated network controls, NAT Gateway or PrivateLink should be reconsidered.

Address Other Fixed Costs

Container Insights has been disabled for both Production ECS clusters, while standard ECS metrics and application logs remain available.

CloudWatch cost $57.10 during the pre-optimization billing cycle. However, the actual reduction from disabling Container Insights can only be confirmed after a complete billing cycle, so we are not treating the estimated reduction as realized savings yet.

The Bastion instance is now started only when needed and releases its public IP when stopped, with estimated savings of approximately $7 per month.

The inactive Worker is expected to save approximately $8–$10 per month.

The two Valkey Serverless instances remain in place. They are already near the minimum storage billing level of approximately 0.1 GB, while their combined ECPU charges are less than $0.01.

Reducing this cost further would require deleting the services entirely, which does not align with current operational requirements.

Change Only One Cost Point at a Time

The entire optimization followed the same validation workflow:

Review current state → Modify one cost point → Roll out the change → Check health → Request the real domain → Review logs and metrics → Remove legacy resources.

If an unexpected replacement or deletion appeared in a Terraform Plan, execution stopped immediately until the resource identity or lifecycle configuration was corrected.

For example, after the Bastion instance was stopped, a state discrepancy caused Terraform to plan an unnecessary recreation of the instance. After correcting the lifecycle configuration and running the Plan again, the changes returned to the expected scope.

Once optimization was complete, Terraform Plan was executed separately for Production and Staging using the correct workspaces and variable files.

Both ultimately returned:

No changes

Final Cost: Optimized Monthly Cost Estimate

The final infrastructure review showed:

One shared ALB remains in each environment;
There are no remaining NAT Gateways;
Production runs four minimum-size online tasks;
One Worker remains configured but stopped;
Production retains one Aurora Writer;
All Staging ECS services are stopped by default;
Staging Aurora can automatically pause when idle.

Cost Item	Estimated Monthly Cost
Production Fargate	$36
Production Aurora, I/O, and Storage	$48–58
Staging Aurora Storage and Minimal Runtime	$2–5
Two Shared ALBs	$35–40
Public IPv4	$29–31
Two Valkey Instances	$12–15
CloudWatch	$3–10
S3, ECR, Secrets, Route 53, and EBS	$7–12
CodeBuild	$5–25
Data Transfer and Other	$2–12

When Staging is rarely activated and builds are infrequent, the estimated monthly cost is approximately $180–$200.

During normal low-load operation, the expected range is approximately $190–$220 per month.

If builds are frequent or Staging is used extensively, monthly costs may rise to approximately $215–$245.

Compared with the full billing cycle before optimization, a typical low-load month is expected to cost approximately 73%–77% less.

The month in which the optimization was performed includes both pre-optimization and post-optimization resource charges, and Cost Explorer data may also be delayed. The actual impact should therefore be evaluated against the next complete billing cycle.

Summary

The core of this optimization was not to reduce every configuration to the absolute minimum. It was to realign infrastructure costs with the system’s current workload while preserving the ability to scale later.

Ingress can be shared. Compute capacity can be reduced based on monitoring data. Test environments can run on demand. Whether an Aurora Reader should remain online depends on both read pressure and recovery objectives. Network egress infrastructure should only be removed after its dependencies have been proven unnecessary.

Five principles are particularly reusable:

Review the bill first, then the monitoring data;
Prioritize fixed costs that are billed hourly;
Migrate and validate first, then remove legacy resources;
Use rolling deployments in Production;
Maintain a clear scaling and recovery path for every reduction in capacity.

Moving from approximately $823 per month to an estimated $180–$220 represents a potential reduction of roughly three-quarters.

More importantly, we now understand why each major cost exists—and which capabilities should be restored first as the business grows.

Data sources and methodology: Costs for the full billing cycle before optimization were sourced from AWS Cost Explorer using UnblendedCost. Resource states, task specifications, and security group configurations were verified through read-only AWS API calls after the optimization was completed. Performance metrics were based on five-minute CloudWatch sampling intervals. Monthly projections use publicly available On-Demand pricing and approximately 730 hours per month. Estimates exclude potential taxes, AWS Support fees, discounts, credits, and unexpected traffic spikes. Project names, domains, account identifiers, and resource identifiers have been anonymized. These estimates do not constitute an official AWS quote.

KV Cache vs. Prompt Cache: What’s the Difference, and How Are They Related?

Jake Tao — Fri, 07 Aug 2026 20:11:14 +0000

This article was originally published on my blog. For the latest version and future updates, please visit the original post:
https://jaketao.com/language/en/kv-cache-vs-prompt-cache/

Every time a large language model generates a token, it draws on the content that came before it. If it had to compute everything from scratch at every step, responses would be much slower. When building an agent, the same set of system prompts, tool definitions, and conversation history is used over and over again. If these were reprocessed each time, latency and computational costs would continually increase.

These two types of redundant computations correspond to two concepts that are often confused: KV Cache and Prompt Cache.

A model's processing of a single request is usually split into two stages: prefill and decode. The KV Cache stops the system from re-doing the work on historical token K/V pairs during decoding, while the Prompt Cache lets later requests reuse the same prefix.

In short:

KV Cache is the underlying state and inference mechanism.
Prompt Cache is the strategy or product capability that reuses these preprocessing results across requests.

A lot of Prompt Cache implementations rely on reusing pre-computed K/V states.

Tip for reading: This article discusses Q, K, V, prefill, decode, prefix matching, and cache breakpoints, also called cache boundaries. You don't need to know anything about math or APIs to understand it. First think of Q, K, and V as "intermediate vectors" in attention calculations. Then follow the two examples: Beijing weather and product manual.

KV Cache: "Intermediate Results" During Model Generation

Large models generate content token by token. Whenever a new token is generated, the model has to consider the tokens that have already appeared.

For example, in a standard Transformer, each token makes three sets of vectors—Q, K, and V—at every layer.

You can think of them as:

Q: "What I'm looking for"
K: "What I have here"
V: "What information I should extract if I'm selected"

In autoregressive decoding, the Q values of historical tokens aren't reused in subsequent steps. However, their K and V values are repeatedly queried by tokens generated later.

So, the model stores these K and V values. This is the KV Cache.

For example, if you were to ask:

What's the weather like in Beijing?

During the prefill phase, the model processes the whole question and stores the K and V values for each token at every layer.

Once the decoding phase starts, new Q, K, and V values are calculated only for the token just added to the sequence at each step.

The model puts the current K and V together with the cached history, then does attention calculations using the current Q on both the historical and current K and V to predict the next token.

This way, you won't have to keep recalculating the K and V of historical tokens.

But that doesn't mean long context is free of cost.

For standard full-attention models, the longer the context:

the more video memory the KV Cache uses;
the more historical K/V pairs usually need to be read at each step.

So, long conversations might still feel slow. Attention structures like sliding windows limit the history that can be seen.

The KV Cache is usually managed by the inference engine, and application developers rarely interact with it directly.

It's mostly used for incremental decoding within a single generation, but the cached K/V state can also be used by the inference framework for cross-request prefix reuse.

The latter is often called a Prompt Cache or Prefix Cache.

Prompt Cache: Eliminating Redundant Processing of Identical Prefixes

When people hear the term "cache," many immediately think of an output cache, where a previously answered question is simply returned.

But the Prompt Cache isn't the same kind of output cache.

Even if there's a cache hit, the model will still regenerate the response.

The Prompt Cache reuses intermediate results from the prefill phase for prompt prefixes, such as K/V states or other similar preprocessing results.

A cache hit:

reduces redundant prefill computations;
shortens the delay for the first token;
may lower the cost of repeated inputs if the API provider charges differently for cached inputs.

For example, let's say you give the model a 50-page product manual and ask:

Product manual → What's the warranty period?

A bit later, you ask another question based on the same manual:

Product manual → What are the requirements for returning an item?

The product manual used in both requests is exactly the same, except for the last question.

If this common prefix is cached, the second request can reuse the preprocessed results associated with the manual and process only the new question that follows.

On the other hand, if you only use this manual once, Prompt Cache might not be that helpful.

For Prompt Caches that use automatic matching or caching based on breakpoints, the reusable portion should typically consist of a continuous, identical prefix starting from the beginning of the prompt.

So, content that changes slowly and can be reused in many ways should go at the beginning, while content that changes frequently should go at the end.

For example:

Long-term stable content:
system prompt, tool definitions

→ Periodically stable content:
user configuration, reference documentation, task background

→ Session content:
conversation history, task status

→ Current request:
current time, temporary information, user question

This isn't a fixed classification.

The key is to arrange content by stability, but this shouldn't alter message roles, command priorities, or business semantics.

Different service providers may use:

automatic matching;
explicit cache breakpoints;
independent cache objects.

Also, keep in mind that minimum length, expiration periods, and billing rules can differ depending on the model.

When integrating, it's a good idea to check the latest model documentation and cache statistics in the response.

Two Common Bad Cases: These Approaches Can Quietly Break Cache Reuse

Here are two common examples.

The code uses Anthropic's cache_control as an example, but other service providers may use automatic matching, different cache markers, or independent cache objects.

So, you can't simply copy these fields across providers.

1. Dynamic Content Can Mess With Prefix Stability

When you're counting on prefix matching, if the content changes at a certain point, the old prefix following that point usually can't be reused.

So, including a timestamp—which changes with every request—in the cache prefix will affect the fixed rules that follow it.

// ❌ Timestamp is included in the cached prefix and changes every request

const system = [{
  type: "text",
  text: `Current time: ${new Date().toISOString()}
You are a code assistant. Here are the fixed behavior rules...`,
  cache_control: { type: "ephemeral" }
}]

A better approach is to put long-term, stable content at the beginning and set a cache breakpoint at the end of the stable prefix.

Dynamic information, like timestamps, should be placed after the cache breakpoint.

// ✅ Stable content first; dynamic content after the cache breakpoint

const system = [
  {
    type: "text",
    text: "You are a code assistant. Here are the fixed behavior rules..."
  },
  {
    type: "text",
    text: "Here are the fixed tool usage instructions...",
    cache_control: { type: "ephemeral" }
  },
  {
    type: "text",
    text: `Current time: ${new Date().toISOString()}`
  }
]

Content like project configurations and reference materials might be somewhere between "long-term stable" and "subject to frequent changes."

You can sort them by stability.

If there are multiple cache breakpoints, set breakpoints for stable prefixes of different lengths.

You also need consistency beyond just the text: the order of tool definitions, image parameters, and other elements may also participate in prefix matching.

2. Only Caching the System Prompt in Multi-Round Conversations

Multi-round conversations usually include the conversation history in every request.

If you set the cache breakpoint only at the end of the system prompt and don't enable automatic caching, the growing conversation history will still need to be processed repeatedly.

// ❌ Only caches the system prompt; conversation history is outside the cache boundary

const request = {
  system: [{
    type: "text",
    text: "You are a code assistant...",
    cache_control: { type: "ephemeral" }
  }],
  messages: history
}

You can use Anthropic's current auto-caching as an example.

Enable cache_control at the top level of the request to automatically move the cache boundary forward as the conversation grows:

// ✅ Automatically cache the prefix of an ever-growing conversation

const request = {
  cache_control: { type: "ephemeral" },
  system: [
    {
      type: "text",
      text: "You are a code assistant..."
    }
  ],
  messages: history
}

Once enabled, the next round of requests can use the prefix that was cached from the previous round.

It processes only the responses, tool calls, tool results, and current question that were added, and writes a new cache prefix for later requests.

So, this reduces unnecessary processing of the conversation history.

It does not mean that only the last message results in a cache miss.

If the service provider doesn't support automatic caching, you need to follow its rules and place an explicit cache breakpoint at a stable position near the end of the conversation.

Setting a cache doesn't guarantee a hit.

When a prefix is encountered for the first time, the system typically has to finish the computation and write it to the cache first.

A cache miss may occur if:

the cache has expired;
the prompt doesn't meet the minimum length;
the historical prefix has changed;
the cache entry isn't yet available.

As of August 2026, each cache breakpoint in Anthropic will only search for previously written cache entries within the most recent 20 content blocks.

A miss may also occur if too many blocks are added during a single Agent cycle.

You can't just look at whether cache configuration is present in the request to determine whether caching really provides benefits.

You should check the cache read, write, and hit metrics that the API returns.

If latency is a concern, you should also log first-token latency on the application side and evaluate cache effectiveness together with actual costs.

Finally, How to Tell the Two Apart

Concept	Core Function
KV Cache (Inference Mechanism)	Reuses the K/V pairs of historical tokens during generation to avoid redundant calculations at each step.
Prompt Cache / Prefix Cache (Cross-Request Reuse)	Reuses prefill results with the same prompt prefix across different requests.

So, the two are not equivalent, nor are they entirely unrelated.

The KV Cache is the underlying state and inference mechanism.

The Prompt Cache reuses the pre-computed prefix state for other requests.

For application developers, the best approach is to keep the common prefix stable and put timestamps, temporary information, and the current question as far toward the end as possible.

References

OpenAI: Prompt caching
Anthropic: Prompt caching
Hugging Face: Caching
vLLM: Automatic Prefix Caching
DeepSeek: Context Caching

From Chat Completions to Responses: Why Is OpenAI Upgrading Its Core API?

Jake Tao — Sat, 01 Aug 2026 06:43:23 +0000

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/why-openai-upgrading-api

If you've ever built a large-language-model application, you've most likely started with this endpoint:

POST /v1/chat/completions

In the era of GPT-3.5 and GPT-4, this endpoint was practically synonymous with the OpenAI API. Developers would pass in a set of messages, and the model would generate the next response based on the context.

But as applications have evolved from "chatbots" to "agents capable of invoking tools, executing tasks, and processing multimodal content," the structure of the API has also begun to change. OpenAI has introduced a more unified approach:

POST /v1/responses

This doesn't mean Chat Completions are obsolete; rather, it provides a more appropriate abstraction for the more complex workflows of agents.

Chat Completions: Conversation Messages at the Center

The core data structure of Chat Completions is messages. In each request round, the client must submit the context required for the model to understand the current task.

For example, a user requests the weather in Beijing:

{
  "model": "gpt-5.6",
  "messages": [
    {
      "role": "user",
      "content": "帮我查询北京天气"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "查询天气",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {
              "type": "string"
            }
          },
          "required": ["city"]
        }
      }
    }
  ]
}

After the model decides to call a tool, it will return a result similar to the following:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_weather_001",
            "type": "function",
            "function": {
              "name": "get_weather",
              "arguments": "{\"city\":\"北京\"}"
            }
          }
        ]
      }
    }
  ]
}

After the application executes get_weather, the next request must include the previous conversation, the tool calls initiated by the model, and the results of those tool executions:

{
  "model": "gpt-5.6",
  "messages": [
    {
      "role": "user",
      "content": "帮我查询北京天气"
    },
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "call_weather_001",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"city\":\"北京\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_weather_001",
      "content": "北京晴，25°C"
    }
  ]
}

This approach is intuitive, well-established, and still suitable for most chat scenarios.

However, it has one obvious engineering shortcoming: context management is primarily handled by the client. As conversations grow longer and tool calls increase, the application must continuously maintain and replay historical messages.

Responses: Centered Around a Single "Task Response"

The Responses API takes a different approach: it treats model output not merely as a piece of text, but as a "response" that may include text, reasoning, tool calls, images, or structured results.

Let's use the weather query as an example again:

{
  "model": "gpt-5.6",
  "input": "帮我查询北京天气",
  "tools": [
    {
      "type": "function",
      "name": "get_weather",
      "description": "查询天气",
      "parameters": {
        "type": "object",
        "properties": {
          "city": {
            "type": "string"
          }
        },
        "required": ["city"]
      }
    }
  ]
}

The model returns a responsecontaining a function call:

{
  "id": "resp_123",
  "output": [
    {
      "type": "function_call",
      "call_id": "call_weather_001",
      "name": "get_weather",
      "arguments": "{\"city\":\"北京\"}"
    }
  ]
}

After the tool completes execution, the next round only needs to submit the new results and reference the previous response:

{
  "model": "gpt-5.6",
  "previous_response_id": "resp_123",
  "input": [
    {
      "type": "function_call_output",
      "call_id": "call_weather_001",
      "output": "北京晴，25°C"
    }
  ]
}

OpenAI can use previous_response_id to associate the previous context with the tool call. The client does not need to manually replay the entire message history each time, making the Agent's orchestration code more concise.

However, note that this does not mean "context no longer incurs costs." Using previous_response_idreduces the complexity for the client in constructing and maintaining the message history; previous input tokens in the response chain will still be billed as input tokens.

Why does the Agent need the Responses API more?

In a question-and-answer scenario, messagesare natural; but Agents often need to constantly switch between conversations, tool calls, tool results, and structured data.

Chat Completions can also handle these tasks, but as the number of steps increases, the client must maintain a complex messages history on its own and ensure that tool calls are correctly mapped to their results.

The focus of the Responses API is not on adding a new capability, but on unifying these elements into response items and supporting the continuation of tasks based on the previous response, making it better suited for complex Agent workflows.

How should an API Gateway be designed?

If the Gateway integrates models such as OpenAI, Claude, Gemini, and DeepSeek simultaneously, the key is not to rewrite all requests as Responses.

A more practical approach is to retain client-familiar interfaces---such as Chat Completions and Responses---for external use; once requests enter the system, they are parsed by the corresponding converters and routed into the same processing pipeline.

OwlVigil adopts precisely this approach: rather than replacing Chat with Responses, it allows different protocols to share the same set of gateway capabilities.

Client
  ├─ Chat Completions
  ├─ Responses
  ├─ Anthropic Messages
  └─ Gemini API
          ↓
      Inbound Converter
          ↓
   Unified LLM Request Model
          ↓
Model mapping, routing, rate limiting, retries
          ↓
       Outbound converter
          ↓
OpenAI
  ├─ Claude
  ├─ Gemini
  └─ DeepSeek

The term "unified" here does not mean forcing a binding to a single vendor's protocol, but rather placing messages, tool calls, tool results, model parameters, and streaming responses into a single processing pipeline.

I’m Finally Building This Full-Time — Meet Syrovex

Jake Tao — Sat, 01 Aug 2026 06:33:10 +0000

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/meet-syrovex/

I have recently embarked on a career as an entrepreneur. I imagine that many of you are thinking, "You just couldn't resist after all!"

During my college years, I often received inquiries from others regarding my decision not to establish a business. I was still relatively inexperienced at the time and was unable to provide a definitive response. On one hand, I could rapidly transform ideas into tangible outcomes, and I found the process of creating something from the ground up to be highly satisfying. However, I did not experience the same sense of urgency to pursue entrepreneurship. Launching a business requires more than just ideas and skill; it often necessitates a certain drive, a catalyst to propel action.

In recent years, I have transitioned between several prominent technology companies, gained extensive experience, and contributed to numerous projects. However, I have not found the professional fulfillment I sought. After deciding last year to take a break and enjoy life, the idea of starting a business did cross my mind, but it was quickly overshadowed by the various demands of daily life. I had assumed that I would continue living a quiet, uneventful life. However, as is often the case, life presented me with an unexpected challenge. During this time, I have read extensively and engaged in profound reflection. The advent of the AI era has given rise to a plethora of new opportunities in the market. When I encounter these opportunities, I find myself contemplating the potential implications. What would be the result if I were to pursue them?

However, launching a business entails more than just pursuing an interest or acting on impulse. Even in the presence of a favorable market and opportunities, the success of a business often depends on the timing, the presence of like-minded individuals, and the identification of a suitable entry point. It was indeed a stroke of luck. In 2026, I had the opportunity to meet with the appropriate individuals at the ideal moment, during which we engaged in productive discussions about the most suitable concepts. Many previously scattered thoughts suddenly coalesced, creating a clear path forward. At that moment, I didn't hesitate long. Instead, I had a clear sense of direction. I understood that the present moment was the right time to take action.

Syrovex

This represents my inaugural foray into full-time entrepreneurship. I have worked on a variety of projects over the years, some of which met with success, while others encountered challenges. However, this is the first time I have devoted all my time and energy to a single company or endeavor. I had envisioned what the circumstances surrounding this moment might be like, but the timing still caught me by surprise.

Regarding the name "Syrovex," there is no particularly noteworthy origin story. This concept was conceived by our co-founders during a recent discussion. While many aspects of starting a business require careful deliberation, naming a company is often the opposite --- sometimes it's just a flash of inspiration in a single moment. When the concept was initially presented, there was a general consensus that it was both pronounceable and visually appealing, leading to its adoption.

The company logo features a four-leaf clover, a symbol of good luck and fortune. In addition to its aesthetic appeal, the symbol represents positive qualities such as fortune, optimism, and progress. The process of establishing a new business enterprise invariably entails a degree of uncertainty. While it is impossible to predict future events with certainty, it is our practice to begin each new undertaking with a modest amount of positive energy.

Why now --- AI as infrastructure, not a tool

As I discussed in "Will AI Replace Software?" and "Several Certain Trends in AI Development Over the Next 6--12 Months," while the development paths of each technological revolution may differ, they often share striking similarities. Whether it's the internet, the First Industrial Revolution hundreds of years ago, or today's AI era, the outlines of many trends were already visible in their early stages.

As large language models continue to mature and stabilize in their development, AI will transition from a mere tool to a new category of "infrastructure" across all industries. Consequently, AI applications will transition from being experimental projects for a select group of technical experts to becoming a daily productivity tool for individuals, teams, and enterprises. This shift is analogous to the transformative impact of the internet and cloud computing. The primary obstacle to widespread adoption of AI is not the capabilities of the models themselves, but rather the governance capabilities. No individual or enterprise would allow such a powerful "weapon" to operate without constraints regarding cost, permissions, security, and compliance. Whether it's an agent built by an individual developer or an enterprise-level workflow, a unified platform is essential for managing, controlling, and auditing the operation of AI before deployment into a production environment. Therefore, Syrovex has chosen to address the most pressing and practical pain points by helping to establish a secure, controllable, and scalable AI infrastructure. This will empower individuals and enterprises to successfully deploy AI applications.

Cost Management

The emergence of the "agent" concept and the viral popularity of "Little Lobster" earlier this year have led to a growing realization that AI is not merely a recreational chat tool, but a productivity tool capable of directly generating value. It is important to note that, unlike the internet, AI does not come free of charge. In the PoC/MVP phase or for personal use, expenses of a few hundred or even a few thousand dollars may seem insignificant. However, when AI begins to integrate into core business processes and is widely adopted by teams, departments, and even the entire enterprise, costs quickly shift from a technical issue to a management issue. Enterprises need to know where their money is being spent, which teams are using AI, which models are the most expensive, which use cases truly create value, and how to prevent budgets from spiraling out of control.

I believe that cost management will be the primary challenge encountered during the implementation of AI applications. Therefore, we have selected AI Cost Management as our initial focus area. This solution empowers enterprises to establish budgets and quotas at the corporate, team, and individual levels, while enabling comprehensive daily, weekly, or monthly management. This allows them to enjoy the productivity gains from AI while maintaining visibility and control over costs.

Global Gateway & Smart Routing

Once AI capabilities reach a stable phase, enterprises and individuals will focus on achieving their business goals in the most stable, efficient, and cost-effective way. The question of which large model to use will become less significant. Current models may be used today, but tomorrow they may be replaced by Claude or Gemini. New models with greater capabilities and lower prices will continue to emerge in the future. This constant switching is extremely painful and troublesome. For most enterprises, models are merely underlying infrastructure; they do not want their business code to require constant adjustments as models change, nor do they want to be locked into a single AI provider.

To address this need, we developed the Global AI Gateway. It is not intended to replace existing AI gateway products; rather, it is designed from the enterprise's perspective to provide a unified AI access layer. This integration enables businesses to seamlessly connect to leading global models and automatically select the most suitable one based on cost, performance, latency, region, availability, and business requirements.

Among its features, I am particularly enthusiastic about Smart Routing. Each large language model has its own strengths, and the most appropriate model for a task depends on the specific requirements. Our objective is to meticulously divide complex tasks into manageable components and allocate them to the models that are most equipped to handle each task. This approach not only optimizes the strengths of different models but also avoids the waste of tokens caused by "using a sledgehammer to crack a nut," thereby further reducing AI usage costs while ensuring effective results.

The Global AI Gateway consolidates access controls, audit logs, data residency, and compliance policies, thereby relieving enterprises of concerns regarding the evolution of underlying models or the vendors utilized. This enables enterprises to prioritize their core business operations with a focus on their own business. By facilitating seamless integration between global AI providers while ensuring security, compliance, and reliability, AI can be established as a stable, sustainably evolving foundational platform, akin to the internet, databases, and cloud services.

AI Governance, Compliance & Visibility

When enterprises adopt AI at scale, they face two main challenges: cost and governance, compliance, and visibility.

In the early stages of AI development, enterprises often experiment on an individual or small-team basis. Employees are able to register for AI services, purchase subscriptions, and integrate different models independently, allowing for rapid validation of various ideas and use cases. While this approach is effective during the exploratory phase, as AI gradually integrates into formal business processes, enterprises begin to realize they are losing their ability to manage AI.

Managers are unaware of which teams are using AI, which data is being sent to external models, or which employees have access to sensitive data. They are also unaware of whether AI usage complies with internal security policies and industry regulatory requirements. Once AI transitions from a personal tool to an enterprise productivity tool, it is essential to have visibility, control, and traceability of users, usage patterns, the data being used, and the resulting outcomes.

Therefore, we have made AI Governance, Compliance, and Visibility one of the platform's core capabilities. Enterprises can centrally manage access permissions for all AI systems within their organization, establish different access policies based on teams, roles, and business scenarios, and record every request and operation through comprehensive audit logs. Concurrently, the platform perpetually monitors AI usage, presenting model usage, cost distribution, access behavior, and potential risks via a dashboard. This enables managers to gain real-time insight into the AI operational status across the entire organization.

I believe that in the future, every enterprise that uses AI at scale will establish its own AI governance system. AI must not only be sufficiently intelligent but also trustworthy. Enterprises that wish to leverage AI as a source of productivity must first develop a comprehensive understanding of AI implementation, including its users and the flow of data. They must also establish processes for rapidly identifying and addressing any issues that may arise.

Workflow Integration

For many individuals and enterprises, contemporary AI remains primarily a tool: launching ChatGPT or another AI product, entering a query, obtaining an answer, and then returning to the original system to resume work. While there has been an improvement in efficiency, AI still operates outside of business processes and has not been fully integrated into the enterprise's production system.

History has shown that significant value has never been created by a single technological innovation. Rather, it has been the result of a comprehensive redesign of business processes that align with the new technology. This is equally applicable to the field of AI. In the future, competition among enterprises will not only be about who possesses the most advanced models. It will also be about who can more quickly and deeply integrate AI into business processes and continuously optimize the way the enterprise operates.

In the current business climate, a growing number of enterprises have come to recognize the significant value of AI. However, there is still a notable gap between recognition and full integration of AI into production environments. The primary impediment to their progress is not a shortage of models, but rather a lack of clarity regarding the initiation of these processes, the identification of which processes are worth transforming, and the integration of AI with existing systems, security requirements, and business logic. Our objective is to assist enterprises in addressing the fundamental challenges associated with AI implementation by leveraging our capabilities in cost management, unified access, governance, and compliance. Building on this foundation, we further provide AI-Powered Workflow Integration services. These services range from business process mapping, process analysis, and AI strategy formulation to the design, development, and deployment of Agents and Workflows. We help enterprises truly achieve their AI transformation.

Agents are merely the starting point for AI implementation; it is in workflows that true value is created. Integration of AI with an enterprise's existing systems, data, and business processes is essential for it to evolve from a supporting tool into an integral part of the enterprise's production system.

If you have already started using AI or are still hesitating about how to take that first step, we invite you to reach out to us. While we may not be able to solve every problem, we can help you avoid some detours and provide a clearer roadmap for AI implementation.

OwlVigil & Agentic Workflow

We named our first-generation product OwlVigil. This name combines "owl" and "vigil." The owl is a symbol of wisdom, insight, and observation, while "Vigil" represents continuous monitoring, protection, and vigilance. This initiative is in alignment with the product's mission to assist enterprises in the management, governance, and protection of their AI usage.

At OwlVigil, our primary objective is to establish a cohesive management and control layer between enterprise applications, tools, workflows, and large language models. We are developing this layer, which we are calling the Enterprise AI Control Layer, to provide enterprises with a secure, controllable, and scalable foundation for leveraging AI. It is now available for use. We warmly welcome your feedback and criticism regarding our product experience, feature design, business model, or our assessment of the AI industry. OwlVigil.com

As stated in our mission: Powering enterprise intelligence at scale. I believe that in the future, every enterprise will utilize AI to some extent. Whether a company is starting with a single business scenario or seeking a comprehensive AI upgrade of its existing processes, we hope to be a long-term partner in their AI transformation journey.

A closing thought, and some good luck

In my professional opinion, the transformation brought about by generative AI is likely to profoundly change the way the entire world operates, much like the birth of the internet. This is not merely a new technology, but rather a new revolution in productivity. This transformation encompasses a wide range of tasks, including knowledge acquisition, content creation, software development, and business operations. In the coming years, we may witness the emergence of a new generation of companies, as well as the complete reshaping of many traditional industries.

It is a great privilege to live in such an era and to be personally involved in this transformation.

This is the first time I've formally shared my entrepreneurial plans with everyone, but I believe it won't be the last. In recent years, I have engaged in a variety of professional activities, including writing code, building products, leading teams, and participating in numerous startup projects. However, this marks the first occasion on which I am assuming the role of an entrepreneur, with full responsibility for the direction of a company, its products, its team, and its future. Many people have expressed to me that embarking on a business venture often entails a significant transition, making it challenging to revert to one's previous lifestyle. I believe I am beginning to grasp the implications of this situation. Entrepreneurship brings excitement and anxiety; a sense of accomplishment, but also countless uncertainties. Precisely because of this, we place great value on each breakthrough, each step of growth, and each moment of recognition from our customers.

We are aware that our journey is far from over and that we will encounter numerous challenges in the future. Regardless of the outcome, I am committed to completing this journey with dedication. I eagerly anticipate the day when I can reflect on my decision to be a part of this transformative era of AI, rather than merely observing it from the sidelines. Syrovex's logo features a four-leaf clover, a symbol of good luck and fortune. While there is no way to guarantee the outcome, it is hoped that, as we navigate this journey filled with uncertainty, it will serve as a source of good fortune, helping us to overcome one challenge after another.

Finally, I would like to express my gratitude to all my colleagues who have provided assistance, confidence, and support throughout my tenure. We should all endeavor to find our place in this emerging field of AI.

A Preview of Key Trends in AI Development Over the Next 6-12 Months

Jake Tao — Sat, 01 Aug 2026 06:04:16 +0000

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/ai-trends-2026

The capital markets' perspective on AI has been quite interesting lately. It began to cool off in the second half of last year, shifted to skepticism about the return on investment early this year, and has recently seen a resurgence of confidence. This is a stark change, and it has been like a roller coaster ride in just a few short months. While concerns remain, the most severe issues appear to be in the past. Regardless of how capital markets view it, AI development follows discernible patterns. By examining historical trends, we can make reasonable predictions about future outcomes, whether they are months or years away. As a professional in the tech industry, I would like to share some insights.

Models are stabilizing and AI applications are booming.

In recent years, many startups have focused on developing AI applications. However, they have encountered challenges as major updates to large language models have rendered their work obsolete. AI applications depend on optimizing and deepening the capabilities of large language models. If these models are upgraded too frequently --- and each upgrade brings a significant leap in capability --- companies will find it difficult to develop applications effectively.

Large language models have rapidly evolved, demonstrating capabilities that have advanced from those of an elementary school student to those of a middle school student and then a high school student in less than three years. It is reasonable to predict that they will soon become "college students," and these "college students" will most likely evolve into "platforms." Subsequent to this period of growth, the development will level off, which will be the ideal time to build AI applications. This trajectory mirrors the precedent set by the early stages of the internet and mobile internet. During the internet revolution of the 1990s, there was a strong interest in showcasing skills, but the period of significant growth and monetization did not occur until 1996 and 1997. It was not until after the dot-com bubble of 2000 that the industry truly began to flourish, and the major companies we recognize today all experienced significant growth during that period. The evolution of mobile internet technology has followed a similar trajectory. After the release of the iPhone in 2007, the market experienced a period of growth and development, leading to the emergence of various mobile applications in 2010. This development phase spanned approximately three years. A significant number of the companies with which we are currently familiar were established around 2010. This pattern is not solely attributable to the cycle of technological development; it is also influenced by the time required for the market to accept new innovations, for startup teams to enter the field, and for investors to adapt their strategies.

The significant popularity of OpenClaw at the beginning of the year demonstrated the market's demand for AI applications, a trend that has become increasingly evident recently as more and more AI applications are being implemented. I anticipate an imminent surge in vertical applications and the emergence of a new generation of startups.

A significant increase in the incorporation of artificial intelligence into business operations

Presently, the majority of AI applications are based on chat windows that operate on a question-and-answer model. While this model may not be the most efficient option, it is designed to address the complex user needs. However, as the field evolves, the demand for AI will become more specific, shifting from solving general problems at a broad level to addressing niche areas. This is analogous to the invention of the steam engine, which generated vast amounts of general-purpose power. However, the subsequent challenge of harnessing that power to solve practical problems required exploration by professionals in various fields.

It is generally accepted that companies can reduce costs by leveraging AI. However, the specific methods for achieving this are not always clear. At this stage, AI cannot function without human involvement. There is currently no indication that it will be able to operate independently in the future. Therefore, it is essential to establish a checkpoint at each work node. If we liken a company's operations to an assembly line, the most practical upgrade would be to divide the line into smaller segments, each with a control center staffed by an operator who verifies the AI's results before proceeding to the next step in the process. The process of segmenting and integrating systems is time-consuming and labor-intensive. However, once established, it will lead to a significant increase in work efficiency and a reduction in the number of employees required.

These enterprise-level services will become the subject of fierce competition among major tech companies. However, as was the case with website development in the past, the market is vast enough to accommodate all participants. Large companies will undoubtedly claim their share, but there will be ample opportunity for small businesses to thrive as well. This trend is expected to persist for several years. In the near future, many labor-intensive tasks will be replaced by AI, and companies will require only a few --- or even just one --- senior staff member to operate the control center.

Token Usage Experience Significant Growth, While Prices Show Decline

The rapid integration of AI into business processes is driving a substantial increase in token usage. This trend is already evident, and I anticipate that this year's surge is just the beginning. However, these measures are still inadequate. This phenomenon might be reminiscent of the early days of dial-up internet, when the amount of data being transferred was unimaginable. Could we have foreseen the immense surge in internet traffic and the significant increase in home bandwidth and internet speeds that we are currently experiencing?

Many people are unaware of the concept of tokens and assume that using AI is cost-free, similar to the internet. However, in reality, AI's token consumption often exceeds expectations. Each interaction and conversation uses hundreds or even thousands of tokens, and these tokens aren't inexpensive. The OpenClaw crayfish bot, which gained widespread attention last month, serves as a prime example. Many individuals enthusiastically implement these systems, subsequently encountering substantial financial surprises.

At present, the use of AI is primarily confined to the domain of "Q&A." Once consumer-facing (2C) applications experience significant growth and business-to-business (B2B) workflows are seamlessly integrated, token consumption will undergo exponential expansion. This should help explain why all the major tech companies are investing heavily in the construction of computing power centers regardless of cost. This will be the biggest growth driver for the next ten or even twenty years. The number of tokens essentially represents computing power, and computing power is fundamentally driven by electricity. Computing power is influenced by the performance of chips. However, electricity is an infrastructure component. In this regard, the U.S. power grid is significantly inferior to China's. However, the U.S. possesses many power-generation technologies independent of the grid, and these technologies will also flourish in the future.

In contrast to the fluctuations seen in a market economy, an increase in token usage will not result in a decrease in its price. As previously discussed, price is influenced by computing power and electricity costs. However, the deployment of data centers and the upgrading of power infrastructure will substantially reduce the cost of tokens, leading to a decrease in their price. In general, the cost of using AI should not increase significantly. This will stimulate the application market, as companies will adopt it to save substantial amounts of money.

From Horizontal to Vertical Development

AI is currently undergoing horizontal development, which refers to the continuous expansion of its general-purpose capabilities. Once it evolves into a platform, numerous vertical development opportunities will emerge. The concept is straightforward: once the platform stabilizes, AI applications are utilized to address a particular class of problems. As the number of applications addressing different problems increases, they will naturally become more specialized. The market determines the demand for these solutions. Customers require services that address their specific needs, rather than a comprehensive but versatile solution.

Presently, the degree to which AI is incorporated into other industries remains quite limited. As AI becomes integrated into business processes, entrepreneurs in various sectors will progressively identify issues that can be addressed by AI, resulting in the formation of more AaaS (AI as a Service) companies. These developments closely mirror the evolution of the internet.

Afterword

I had a sudden flash of insight this evening and would like to share my thoughts. This world is essentially a makeshift operation; for many things, once you grasp their essence, the path forward becomes crystal clear. However, this knowledge alone is not sufficient for success. External factors such as timing, motivation, and luck play a crucial role in determining outcomes. The unpredictability of our world is a source of fascination, but it also leaves us with a sense of regret.

Will AI replace software?

Jake Tao — Sat, 01 Aug 2026 05:48:16 +0000

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/will-ai-replace-software/

The concept that "AI will replace software" has recently gained significant traction. This has led to a significant decline in software stocks, and when combined with concerns about the profitability of AI, has caused widespread anxiety in the market. While there are legitimate concerns about the profitability of AI, it is important to note that the cost of developing AI remains high, and revenue models continue to rely on traditional methods such as advertising and recommendations. The assertion that AI will disrupt the software industry may be somewhat exaggerated.

This discussion takes place against the backdrop of a recent surge in groundbreaking AI applications. First, the sudden rise in popularity of Clawbot allowed users to build personalized AI systems on their local devices, enabling automated operations and process execution --- essentially functioning as a "personal assistant." Subsequently, Anthropic released Cowork, further enhancing agents' ability to perform complex tasks and demonstrating AI's potential for cross-process collaboration. These developments send a clear signal: AI is gradually transitioning from an auxiliary tool to a system application with executive capabilities. In the near future, it is entirely foreseeable that AI will integrate with personal devices and data resources, allowing users to simply issue commands for AI to invoke various processes to complete the required tasks. This evolution will see AI evolve from its current role as an auxiliary "secretary" into a "hands-on" and capable assistant.

A more critical issue lies in the fact that certain software may no longer be considered an essential necessity. In AI-driven workflows, systems will rely more heavily on callable, specialized functional modules. In simple scenarios, AI can dynamically generate or temporarily build the required capabilities. From this perspective, this will certainly have a substantial negative impact on the software industry.

What impact will AI have on software?

Before drawing conclusions, it is essential to examine the substantive impacts that AI's development will have on the software industry.

Software Development Becomes EasierIt is generally accepted that software development has become easier. Historically, the development of software has been a time-consuming process for teams. With the use of AI, a simple program can be completed by just a few senior engineers handling architecture design, code integration, and final debugging. For example, Clawbot was built by a single person, though this analogy isn't entirely accurate. However, it clearly demonstrates how AI can greatly enhance a programmer's development capabilities. While AI is not yet capable of writing highly complex programs, it is possible to break a program down into smaller components. These components can then be debugged and integrated by experienced engineers. This approach has been shown to significantly reduce the software development cycle and lower development costs.

What are the potential implications of this decision?

First, it should be noted that plug-in and utility-type software programs, which are typically billed monthly, may become obsolete in the future. This is due to the fact that creating a plug-in with AI is a straightforward process, and AI can also be customized to specific situations. As a result, these solutions often align more closely with a company's business logic than off-the-shelf plug-ins currently on the market.

Secondly, there will be an increase in the availability of affordable alternatives for large-scale software. While it may not be feasible to develop a comprehensive, large-scale software application from the beginning, the integration of AI can implement specific functionalities, thereby empowering users to achieve their objectives without the need to rely exclusively on a particular program. This may prompt some light users to explore alternative options.

Disruption of Pricing Models

At present, almost all SaaS pricing models rely on a combination of monthly and per-user fees. However, this model may not be sustainable. As previously indicated, users will become less dependent on basic utility software, and the number of light users will also decrease. In such cases, the monthly billing model becomes financially disadvantageous for users. Given the model's previous unpopularity, users now have the opportunity to avoid it. It is likely that they will disseminate information about this fact.

Furthermore, per-user billing will become increasingly challenging as AI enhances work efficiency, allowing tasks that previously required significant manpower to be handled by just a few, or even a single AI "user," with the rest accessing the entire software suite through that AI machine or agent.

In this scenario, software companies will need to adapt their business models and explore new approaches, such as charging based on features and usage frequency (similar to a token system).

Redesigning Software Architecture

Many large-scale software applications nowadays provide comprehensive features and sophisticated interactions. However, as AI technology continues to evolve, users may find that only a portion of these features is necessary. This is due to the fact that AI is capable of breaking down tasks and planning and executing workflows in the most efficient manner. For AI, large software applications consume resources differently than smaller functional modules; it will inevitably choose the workflow that consumes fewer resources. This will compel prominent software companies to modularize their software in order to retain substantial AI users.

This aligns with user logic as well. While many software applications are very comprehensive, most users only utilize a portion of their features. Existing pricing strategies are mostly designed around these popular features.

Shifting Paradigms in User Interface Design

Presently, software interfaces with users through its front end, whereas AI merely needs to call APIs --- adopting a wholly distinct approach. The design and development of these front ends can be a time-consuming and labor-intensive process, especially given the need for distinct designs to meet the diverse needs of different customer segments. This complexity makes software replication a challenging endeavor. However, this will inevitably lead to a decline in its value. In the context of AI, the primary requirement is for the system to comprehend the API in order to operate effectively. The software's front end will no longer be visible to users, and the complex buttons and menus will be removed.

App Store?

In the near future, AI may emerge as a pivotal interface for user interaction, similar to the transformative impact of the iPhone on the app era. It is not yet clear whether a model similar to the App Store will emerge. I believe there is a strong possibility that this will happen. If AI becomes a gateway, all software accessible through that AI will require platform authorization, and users will need to purchase the software or features they need through the platform to build a personalized AI workspace. Consequently, major AI platforms will possess substantial bargaining power, potentially imposing a "tax" on software listings (akin to the "Apple tax"), further reducing software companies' profits.

Advantages of existing software

For complex, dynamic, and large-scale software --- whether engineering or industry-specific --- it is almost impossible for AI to mimic or replicate it. AI is adept at understanding a user's clear objective, breaking it down, and executing each step in a sequential manner. However, when dealing with complex workflows and environments, users may face challenges in accurately articulating all the circumstances, thereby hindering the effectiveness of AI.

This is also the greatest advantage of most SaaS companies. Users could break down the entire process into individual tasks and have AI execute them step by step. While this is not difficult to achieve, it would increase the user's time cost. After carefully evaluating the advantages and disadvantages, users may still decide to utilize professional software. To illustrate with a practical example, consider the annual tax filing process. The forms are available to the public, and the instructions for filing are clearly outlined. However, individuals often prefer to allocate their financial resources towards acquiring software or engaging the services of an accountant to manage their finances, rather than undertaking these tasks themselves. In the context of AI, filing taxes is a straightforward, single task, and I believe AI will be capable of handling it in the near future. However, organizing all your tax forms and submitting them to AI is a complex process. In this regard, software companies have a significant advantage. If they decide to develop distinct AI functions or incorporate AI capabilities into their own software, it will significantly boost its competitiveness in the market. Would you prefer to invest 50 yuan in tax filing software equipped with AI capabilities, or to personally organize your materials and educate the AI to perform the task systematically? Furthermore, if you are responsible for preparing your own taxes using an AI file, who will be held accountable for the results? If there is a mistake, it could lead to further complications.

Consequently, software companies may enhance their competitive edge by leveraging AI, thereby streamlining user tasks. If I were to build a boat, I might feel confident, but when it comes to building an aircraft carrier, I'd still worry about whether the internal systems are fully developed, despite how similar it looks on the outside. Instead of expending valuable resources on this endeavor, it would be more prudent to invest in a solution that can deliver tangible results. In essence, software companies enable collaboration and cost sharing for the utilization of their "aircraft carrier" model. This is also the approach many SaaS companies take today: I understand that you have the capacity to solve the problem independently, but I can assist you in achieving a more efficient solution. This will free up your time, allowing you to focus on your core business operations.

Will software be replaced by AI?

The answer is unequivocally "no." If you reflect on the aforementioned analysis, you will see that while AI has a significant impact on software companies, the fundamental principle remains unchanged: as long as AI cannot independently and efficiently develop complex software, it will always require software to perform the work for it. It is not practical for AI to develop complex software because it still lacks a comprehensive understanding of complex workflows and the human world. Bridging these gaps through self-learning is a significant challenge. If AI becomes capable of handling these tasks, I believe the problems we'll face will be far greater than simply software being replaced.

The fact that software won't be replaced by AI doesn't mean all software will survive; what AI is disrupting are the software industry's profit models, use cases, and business logic. The true value of the SaaS model lies in its ability to manage complex processes, a capability that AI will not be able to provide in the near future. Smaller software applications, particularly utility tools, will be significantly impacted.

In summary, we are currently discussing off-the-shelf software, and software companies already possess strong technical teams and development capabilities. It is conceivable that they would adopt AI to transform their software into an AI gateway. It is my understanding that the interaction model, which currently requires users to find and click buttons, would be transformed into a dialogue with AI. This would reduce the cost of using and learning the software, thereby increasing user reliance on the software itself. Take Photoshop, for example. The learning curve is quite steep, but users can articulate their goals with clarity. It would be worthwhile to consider the development of an AI interface, whether in desktop software, web browsers, mobile applications, or the cloud. This interface would allow users to simply issue commands, and the AI would complete the image edits. For power users, a semi-automatic mode might be ideal. They could assign clearly defined tasks to the AI and handle the rest themselves. This approach may be more appealing than a fully automated model that relies entirely on third-party AI.

AI Is Accelerating the Divide Between People

Jake Tao — Tue, 23 Jun 2026 05:38:45 +0000

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/ai-exposure-gap-between-people

In 2026, generative AI (GenAI) gradually evolved and took shape as so-called "agents," entering everyone's lives at an astonishing pace (regardless of the specific concept, this article collectively refers to them as "AI" for the sake of simplicity). From content creation and information retrieval to daily office work, an increasing number of professionals are turning to AI to enhance efficiency and support decision-making processes. The integration of AI has enabled the rapid transformation of abstract concepts into tangible realities. Software development is a relevant example here: in the past, many people had a wealth of creative ideas and product concepts, but were unable to bring them to life due to a lack of programming skills. However, the advent of AI-powered coding has effectively lowered the barriers to entry across the entire process, from requirements specification to code generation and deployment. In today's digital landscape, developers have the ability to rapidly create product prototypes and entire applications based solely on conceptual ideas.

Consequently, disparities between individuals have also widened. For some, AI serves as an accelerator of efficiency; for others, it's more like a shortcut that bypasses the process of learning and accumulation. AI has the potential to empower individuals to achieve tasks that were previously challenging, but it also brings to light issues that were once difficult to identify.

Ownership: The strong tend to become more productive, while the weak become less active.

Laziness is an inherent aspect of human nature, which is why ownership is of particular importance. In the AI era, what is truly scarce is not the act of getting things done itself, but rather judgment, critical thinking, and a sense of responsibility for the results. Those who lack ownership often merely "check off tasks."

To illustrate, consider the following example. Your superior requests that you conduct industry research or compile a set of materials. Some individuals may opt to delegate this task to artificial intelligence, refining it slightly before generating a comprehensive report with a well-designed structure, sophisticated formatting, and professional language. The document appears to be of a high caliber, potentially surpassing the quality of many individuals' own work. This can lead to a tendency to skim through the material and submit it without thorough review.

However, it is important to assess whether such an outcome is truly valuable. While the content generated by AI is often accurate, it frequently lacks the critical element of thought. AI is not equipped to determine the rationale behind a particular need for information, identify the challenges currently being faced by the team, or ascertain the true company priorities. While the answers it provides may be comprehensive, objective, and logically sound, they could also be a mere pile of "correct nonsense." This is due to the fact that AI is particularly adept at answering questions but lacks the capacity to effectively solve problems. This observation aligns with the concept of "talkers," as discussed in the article "Stay Away from Those Who Just Talk." While many people are impressed by the results generated by AI, which are often considered to be flawless or to exceed expectations, the true value of these results is yet to be determined.

This impression arises because the content remains at the informational level without addressing the actual problem. A person who truly takes ownership of a task does not first think about "how to complete it" when it is assigned, but rather "why it needs to be done." To determine the appropriate angle for conducting research and identifying areas requiring the most attention, it is essential to thoughtfully consider these questions. They will break the problem down into multiple parts, develop their own analytical framework, and then use AI to verify, supplement, and challenge their own judgments. This process is significantly more time-consuming than simply querying AI, and the value of the final output is entirely different.

What is the rationale behind the boss's need for this information?
What are the team's current challenges?
What decisions will this information ultimately influence?
Which details are essential and which are actually irrelevant?

This dynamic has led to a noteworthy development in the AI era: the prosperous are becoming more productive, while the less fortunate are becoming more leisurely. The company has adopted a strong use of AI to transform its work methods, breaking down, optimizing, and enhancing tasks that were previously impossible to complete due to time and cost constraints. The advent of AI has not diminished their intellectual capacity; on the contrary, it has enabled them to delve more profoundly into complex issues. Conversely, underperformers often approach AI with a superficial mindset, perceiving it merely as a solution provider. While it may seem that they are completing more tasks, they are actually just reducing their own thought process. For these professionals, the time saved by AI merely allows them to complete work faster, shifting the responsibility of thinking to others.

AI has not reduced the workload; it has merely shifted the focus from the execution level to the thinking level. This will also become the standard for screening and weeding out talent in the next era.

While AI-generated work is indeed identifiable at first glance, this is not the primary concern.

Many people are perplexed by this phenomenon. I would like to inquire about the reason for my boss's displeasure upon reviewing the polished report that was generated by artificial intelligence. Could his concerns be related to my use of AI?

In most cases, the boss is not concerned with the authorship of the AI-generated content. He is upset because you did not dedicate yourself fully to the project.

To you, this might seem like a high-quality piece of work --- visually appealing, well-structured, logically sound, and even accompanied by charts and data analysis. However, even after devoting ten or twenty minutes to meticulously reviewing numerous pages, it can be challenging to identify truly valuable information, which is often obscured by a substantial volume of professional text. While AI has proven to be highly effective in generating content, it lacks the capacity to discern what is truly important or to determine which approaches are practical and actionable. Consequently, individuals with a deficiency in critical thinking often find themselves susceptible to pitfalls, such as equating ostentatious content with value and misapplying the concept of comprehensive expression to imply profound comprehension. The end result is a report that "looks professional" but is full of "nonsense."

What is the conclusion?
What are the reasons for this occurrence?
Which risks pose the greatest threat?
What are the next steps?

The "One Pager," a term coined by Amazon, is a common requirement in business projects and reports, emphasizing the need to convey information succinctly on a single page. If the core points cannot be distilled onto a single page, including additional content will not improve the document. This is analogous to the one-page limit for resumes.Additionally, when highly skilled individuals utilize AI, their deliverables often become more concise. They utilize artificial intelligence to gather information, validate hypotheses, and expand their thinking. However, when they present their findings to decision-makers, they often limit the information to just the most important conclusion.

As barriers are eliminated, disparities begin to manifest.

AI is rapidly transforming the landscape of knowledge and skill sets, but this evolution does not guarantee equitable outcomes. Instead, it is likely to exacerbate existing disparities in wealth and opportunity. Historically, the development of software necessitated the expertise of programmers, financial analysis required a background in finance, and the design of marketing strategies demanded marketing experience. In today's business world, artificial intelligence (AI) has become a valuable asset, allowing finance professionals to develop apps, product managers to write code, and engineers to quickly learn about investing, law, or even marketing. The cost of acquiring cross-disciplinary skills is decreasing. Could this imply that industry experience and professional expertise are no longer significant factors?

On the contrary, the evidence suggests that the opposite is true.

Historically, the disparities among individuals primarily stemmed from limitations in accessing information, knowledge, and skills. With the significant advancements in AI technology, these barriers have been lowered, allowing other factors to come to the fore. These include judgment, depth of understanding, and the quality of thinking. In summary, AI has reduced the competency gap but increased the cognitive gap. While AI can assist in coding, it lacks the capacity to evaluate the viability of a product for development. Similarly, while AI can aid in generating a business plan, it cannot ascertain market demand. Additionally, while AI can support in data analysis, it cannot determine the significance of data or the reliability of conclusions.

Take AI Coding, for example. In today's business world, professionals in fields such as product management, operations, and marketing can utilize tools like Claude Code and Codex to expedite the development of websites, systems, and the deployment of products to production environments. This is an exciting development. The true challenge, however, lies not in the construction of the product itself, but rather in the creation of a product that truly meets the needs of its intended users. While AI can assist in implementing features, it does not proactively question requirements. It aids in problem-solving, but does not indicate whether a problem is worth solving. While it provides answers, it rarely alerts users to potential flaws in the problem itself. Its effectiveness lies in its ability to provide answers to inquiries rather than defining the terms itself.

While AI coding can facilitate development, it does not inherently reduce the complexity of the system itself. The operational readiness of a demo is not necessarily indicative of its ability to support real users, and the long-term stability of a feature is not guaranteed by its initial implementation. From access control and data security to performance optimization, monitoring and alerts, disaster recovery, cost control, and compliance requirements --- not to mention the various edge cases and anomalies that may arise as the user base grows --- these issues won't automatically disappear just because AI is involved. Often, the most challenging aspect of software development is not the writing of code, but the anticipation of problems, the design of solutions, and the making of trade-offs under various constraints.

Despite the integration of artificial intelligence (AI), certain individuals will likely demonstrate heightened proficiency, while others may face challenges in identifying their needs and selecting the appropriate keywords to effectively convert into prompts for AI-driven processing.

AI will facilitate the initiation of many individuals, yet advanced expertise and comprehension will persist in their scarcity. As the number of professionals in a given field increases, the focus shifts from the question of whether one can do something, to whether one has the necessary understanding, and from whether one can build something, to whether one can build it correctly.

Soul

I have selected "soul" as my concluding point because, in the current era of artificial intelligence, its significance has increased to an unprecedented degree. Today's AI systems are designed to learn from and synthesize the knowledge and experience accumulated by humans over time. These systems can imitate, integrate, and generate, but they lack the capacity to possess their own values, beliefs, or aspirations. The true essence of a work is not found in the tool itself, but rather in the individual who wields it.

AI has the potential to make knowledge readily accessible and execution more efficient than ever before. However, it cannot replace the experience and insights accumulated by a professional over many years. It is essential to emphasize that a sense of responsibility, judgment, values, and accountability for outcomes is not only indispensable but will be significantly amplified.