DEV Community: Truman

From the Fantasy of Local Models to a Truly Controllable Sports Agent

Truman — Wed, 08 Jul 2026 14:14:27 +0000

Looking back, this project did not begin as a fully designed AI agent project. In fact, I had not even decided from the outset that I wanted to build a sports question-answering system.

The original idea was much simpler.

At the time, I frequently saw people on X sharing their experiences with locally deployed large language models. Local AI was advertised as having many advantages: data could remain on your own device, privacy requirements could be satisfied, you would not be restricted by the service policies of major model providers, and you would not have to worry about your account, API access, or region suddenly being restricted one day.

The models might not be as capable as the best cloud-based models, but as long as they were good enough for everyday use, the idea still seemed highly attractive.

I had also been following Mr. Wang from LazyCat Microserver for some time. He promoted the company’s products almost every day, which sparked my interest in the LazyCat AI Pod. Eventually, I ordered one.

After receiving the device, I began experimenting with various models that could be deployed locally. Following a round of comparisons, I ultimately chose the DeepSeek R1 distilled version of Qwen 32B.

To extract as much performance as possible from the hardware, I repeatedly adjusted the model’s startup parameters based on recommendations from GPT. I pushed VRAM allocation, context length, concurrency, and inference settings as close as possible to the maximum that the hardware could handle at the time.

When I first got the model running, I was naturally a little excited. After all, a model with tens of billions of parameters was genuinely running on a device that I owned. It felt as though I finally had an AI that belonged entirely to me.

But after using it a few times, my most immediate reaction was:

Fuck, something was wrong.

Its responses were slow, and its knowledge was outdated. More importantly, there was an enormous gap between it and the GPT experience I was used to when it came to understanding questions, maintaining context, organizing answers, and handling complex tasks.

It was no longer simply a matter of being “slightly less capable” or “slightly slower.” They were not even operating in the same dimension.

That was the first reality I encountered during this project:

Being able to run a model does not mean that the model has practical value.

Local deployment solves questions about where the model runs, who controls the data, and how dependent you are on a vendor. It does not automatically solve problems involving model capability, knowledge freshness, contextual understanding, reasoning quality, or response speed.

When the model itself is clearly not capable enough, the idea of “local AI freedom” can easily turn into an expensive, slow machine that produces answers that are not particularly useful.

Starting a Chatbot as a Way to Make Use of the Hardware

Since I had already bought the device and successfully deployed the model, I decided to make some use of it by building at least a simple chatbot.

At first, I did not seriously consider whether it had the conditions required for real-world operation. I simply added several prompt chains around the model, hoping that breaking tasks into multiple prompting stages would compensate for the model’s limited capabilities.

But I quickly realized that building a general-purpose chatbot was pointless.

It could never be more general than cloud models such as GPT or Gemini, nor could it compete with them in knowledge, reasoning, or response speed. Once I built it, even I found it frustrating to use.

A general-purpose chatbot that I did not want to use myself was not worth continuing to develop.

So I narrowed the scope and decided to build a sports scores chatbot.

Compared with open-domain question answering, sports scores at least offered a clearly defined data scope, fixed rules, queryable interfaces, and relatively well-defined question types. From a product-scope perspective, it seemed more suitable for a limited local model.

However, relying solely on an LLM was obviously insufficient. The model did not know the latest scores, and it should not be expected to explain every sports rule from its own memory.

Following guidance from GPT, I began collecting rules and documentation for major sports and esports. I organized the material into YAML files, built a vector index with FAISS, and integrated it into the question-answering workflow.

This was probably my first real exposure to the idea of retrieval augmentation.

My understanding at the time was very straightforward: if the model did not know something, search the knowledge base first, then insert the retrieved results into the prompt.

The idea itself was not wrong, but after running the system a few times, the problems quickly became apparent.

Each vector search returned a large volume of rules data. That content could consume roughly 80% of Qwen 32B’s available input space. The remaining context might only be enough for the user’s first question.

By the second turn, once the conversation history, rules content, and previous answer were all included, the system could easily exceed the context limit.

That was when I truly realized:

Context is not a warehouse. It is a highly limited runtime budget.

Retrieval does not mean stuffing every piece of “potentially relevant” information into the model. More retrieved content does not necessarily produce a more accurate answer.

Large amounts of low-density information not only consume context, but also dilute the genuinely important details, making it even harder for an already limited model to identify what matters.

Although I was already using RAG and vector retrieval, I had only completed the step of “search for it and put it into the prompt.” I had not yet truly understood context engineering.

The Second Collision with the API Knowledge Base and Vector Retrieval

As the project continued, I began adding more sports scores APIs.

These APIs were not designed specifically for LLMs. Some were used to search for competitions, while others queried fixtures, standings, lineups, match events, team information, or player details.

Many of these APIs had explicit hierarchical dependencies: the output of one endpoint served as an input parameter for another.

To help the system understand these interfaces, I gradually created an api_kb.yaml file containing API descriptions, input parameters, output fields, enumerations, data relationships, and usage restrictions.

Since I had already used FAISS, I naturally applied the same idea to the API knowledge base. I hoped that vector search could identify the APIs relevant to a user’s question, together with their query rules and required execution order.

At the time, I assumed that if the API documentation was sufficiently complete and vector search could retrieve semantically similar endpoints, the LLM would be able to combine those endpoints on its own.

The actual results were poor.

FAISS could find API documentation that appeared relevant, but it could not reliably understand the real business dependencies between interfaces.

It might retrieve an API with a similar field name while ignoring the fact that one of its parameters first had to be obtained from another API. It might also mix together interfaces intended for teams, competitions, seasons, or match details.

More importantly, the retrieval results themselves were unpredictable.

The same question could return different documents at different times. A slight change in intent could cause the retrieved results to drift away from what was actually needed.

The final experience was that the system sometimes worked and sometimes failed completely, with no reliable way to predict which outcome would occur.

The user experience was terrible.

At the time, however, I did not abandon vector retrieval. I simply continued adding rules and prompts around it, adjusting similarity thresholds, and changing the number of retrieved documents, hoping that further optimization would solve the problem.

Looking back, I was asking a retrieval system to take responsibility for something it was not good at.

Vector retrieval is suitable for finding related knowledge. It is not suitable for independently deciding how a group of APIs with strict parameter dependencies should be executed.

Knowledge relevance and tool executability are two entirely different problems.

Multi-Model Routing Did Not Solve the Architectural Problem

During this process, I also integrated the Gemini API. My goal was to classify requests by complexity and route them to different models.

Simple tasks could be handled by cheaper, faster models, while more complex tasks would be sent to more capable ones. In theory, this would reduce costs while improving overall response time.

Later, I also integrated the LLM APIs provided by GitHub and the Grok API, further expanding the routing system in the hope that different providers could complement one another.

However, LLM calls involve many unpredictable situations, particularly when local models are involved.

Sometimes a request would remain unanswered for a long time. Sometimes the returned format would not meet the requirements. Sometimes the model would misunderstand the intent. Sometimes the generated JSON could not be parsed. Sometimes the service was already in an unhealthy state, while the upstream workflow continued waiting.

I therefore began assigning different timeout limits to different models and added service health checks, exception handling, response-format validation, and fallback mechanisms.

When a cheaper model failed, the request would switch to another model. If that model also failed, it would switch again.

These features did improve availability, but they also helped me understand something:

Model routing can address cost and some reliability issues, but it cannot fix an incorrect system architecture.

If intent recognition is unstable, API planning is fundamentally flawed, and the input-output boundaries are unclear, adding more models simply passes the same problem from one model to another.

The Decision Flow and the Debt Created by Keyword Matching

The original workflow included intent detection, rule retrieval, API retrieval, API execution, and final answer generation, but it lacked a node that was genuinely responsible for business planning.

I therefore decided to add a “decision flow” node.

This node needed to select the appropriate scores APIs based on user intent, process API inputs and outputs, understand dependencies between different interfaces, and determine which calls needed to run sequentially and which could run in parallel.

The direction itself was correct.

The problem was the implementation.

While building out the decision flow, Codex chose the easiest and most direct development approach: identifying user intent through keyword matching.

If the user mentioned “standings,” the system selected the standings API. If the user mentioned “lineup,” it selected the lineup API. If a particular fixed term appeared, the request entered a corresponding fixed workflow.

With a small number of test cases, this approach appeared highly effective. The code was generated quickly, and the functionality could be made to work almost immediately.

But as the range of question types expanded, the keyword rules grew rapidly.

The same word could express different intents in different contexts. Different languages, aliases, natural expressions, and conversational follow-up questions could never be exhaustively covered through a finite list of keywords.

When the user said “What about this match?”, “The second one,” “The one from earlier,” or “This Portugal match,” the system did not need to understand an isolated keyword. It needed to understand the current conversational focus, the candidate list from the previous turn, and the goal the user was continuing to pursue.

Once keyword matching entered this kind of scenario, it became increasingly difficult to maintain.

To patch these problems, the decision flow gradually accumulated conversational disambiguation, candidate matching, returning to the original question after candidate selection, user-focus redirection, and new-topic versus old-topic detection.

The number of features continued to grow, and the code became increasingly bloated.

At the time, I was still relying heavily on Codex for vibe coding. As long as the current tests passed, I deliberately ignored code structure and long-term maintainability.

Those problems did not disappear.

They were merely postponed.

Sports APIs Were Not Designed for LLMs

Another source of complexity was that the sports scores APIs used by the project were never designed to be called by an LLM.

Many endpoints required internal IDs, but ordinary users only referred to team names, league names, or particular matches. They did not know, and should not need to know, values such as matchId, leagueId, seasonId, teamId, or playerId.

Before calling the actual business API, the system therefore usually needed to call a search API first, converting a team, competition, or player mentioned in natural language into an internal object.

When the candidates were not unique, the system needed to present them to the user for selection. After the user selected one, the system then had to return to the original question and complete the original task, rather than stopping at “You selected the second item.”

API responses could not be passed directly to the model either.

Many endpoints returned large numbers of fields containing null, internal statuses, irrelevant media URLs, duplicated structures, and data that had no value for the current question.

Without preprocessing, the response from a single API could consume a large amount of context.

As a result, the project accumulated an increasing number of pre-call and post-call processing steps: parameter completion, contextual binding, dependency-parameter extraction, removal of null-valued fields, list truncation, field projection, and response normalization.

I gradually realized that allowing an LLM to call tools was not as simple as asking the model to generate an API name and a set of parameters.

The genuinely difficult part was establishing a stable boundary between the world of natural language and the world of internal APIs.

Deliberately Narrowing the Scope

As development continued, I realized that I could not support every sport at the same time.

Different sports have different competition structures, match states, rules, statistical fields, and APIs. Mixing football, basketball, esports, and other sports together would cause the intent taxonomy, data structures, and testing scope to expand without limit.

I therefore modified the user-intent detection node so that only football and basketball questions were allowed to enter the internal sports API workflow. Esports and other sports were rejected directly.

On the surface, this reduced the product’s capabilities.

In practice, it was the first time the project established a genuine product boundary.

A broad system whose correctness cannot be guaranteed is less valuable than a limited system that can be explained, tested, and controlled.

The project later defined its priorities as follows:

Accuracy over controllability, controllability over speed, and speed over completeness.

This principle was not designed at the beginning. It was a conclusion reached only after a large number of failures.

From “Continuing to Patch” to Redesigning the System

The project truly began to change when I started reading Google Cloud’s materials on AI agents and learning more about agentic design patterns.

Before that, I already had prompt chains, intent detection, API calls, contextual state, candidate selection, and model routing. But these elements were merely accumulated features. They did not form a system with clearly defined responsibilities.

After reading those materials, I began to understand agents differently.

An agent does not mean allowing a model to decide everything freely. Nor does it mean placing all data, tools, and rules into a prompt at once.

A genuinely controllable agent system must separate uncertain judgment from deterministic execution.

LLMs are good at understanding natural language, identifying user goals, handling ambiguous expressions, generating plans, and organizing answers.

Deterministic code is better suited to checking parameters, validating dependencies, restricting the available toolset, binding internal IDs, executing APIs, and blocking invalid plans.

Based on this understanding, I redesigned the entire project.

I disabled rule retrieval by default.

When a user provides an exact score and asks the system to ground the answer, a passage explaining football rules cannot serve as evidence for that score.

Match facts should come from real-time APIs. The rules knowledge base can explain rules, fields, and enumerations, but it cannot be treated as evidence for live match facts.

I also gradually abandoned the practice of integrating large numbers of models to keep the workflow running. I switched entirely to Gemini and divided the runtime into two modes: normal and optimal.

Both modes use the same business workflow and safety boundaries. They differ only in the capability and cost of the models used at different stages.

In this design, differences between models no longer change system behavior. They only affect execution quality and cost.

The final system was not a freely acting autonomous agent. It was a controlled agentic workflow.

A user question first passes through goal monitoring, intent understanding, and contextual resolution. A planner then generates a structured plan.

The Tool Graph Validator checks whether the requested tools exist, whether their parameters are available, and whether their dependency chains are complete.

The Tool Executor calls the internal sports APIs. The Web Evidence Agent handles official sources and general web searches.

Reflection determines whether the available data is sufficient and, when necessary, permits only one repair attempt.

Finally, the Answer Composer produces a unified response, while grounding checks restrict the permitted sources of factual claims.

Within this architecture, the LLM can perform the tasks it is good at, but it does not have the authority to bypass the validator and execute tools arbitrarily.

Only then did I finally understand that the “intelligence” of an agent should not come from giving the model unlimited freedom.

It should come from this principle:

Within clearly defined boundaries, leave uncertainty to the model and certainty to the program.

Web Evidence Changed How I Understood “Internet Access”

Later in the project, I added Web Evidence.

When I originally built the rules knowledge base and FAISS retrieval, my understanding of external knowledge was still based on traditional RAG: collect content, build an index, perform vector retrieval, and insert the results into the prompt.

After the redesign, however, I did not turn Web Evidence into another permanent vector database.

The current workflow only runs official_source or general_search when the LLM planner determines that the user’s question genuinely requires external evidence.

Official sources are primarily used for websites such as FIFA and UEFA. General search is used for news, injuries, transfers, historical context, and questions that cannot be covered by the internal data.

Web Evidence is also not allowed to generate the final answer directly.

It can only return a structured Evidence Pack containing sources, evidence, conflicts, failures, and timing information. The unified Answer Composer must still generate the final response by combining that evidence with data from the internal APIs.

When internal sports APIs conflict with external official sources, the system cannot simply choose one and overwrite the other. The conflict must be explicitly preserved.

While reverse-engineering the UEFA and FIFA websites, I also gained a stronger appreciation for the advantages of AI when handling large codebases and web resources.

The API requests used by these official websites involved many complex restrictions, and their frontend JavaScript files were extremely large.

However, once the relevant JavaScript files were provided to an AI, it could quickly locate the required keys, endpoint URLs, and parameter logic through search.

Whether a file contained ten thousand lines, one hundred thousand lines, or even more made little fundamental difference to the AI.

This is one of the most powerful aspects of AI-assisted programming.

But the same capability also concealed another problem.

AI Can Read One Hundred Thousand Lines of Code. Humans Cannot.

Once the project’s functionality was largely complete, I began seriously reading the code that Codex had generated for me.

The experience was extremely painful.

Thousands or even tens of thousands of lines of code had been concentrated into individual files.

For an AI, this was not a problem. It did not need to read the code from the first line to the last in the way a human would. It could search directly for the relevant locations through commands, symbols, and keywords.

To an AI, the difference between one hundred thousand lines and ten thousand lines might simply be the size of the search space.

But that was not true for me.

I needed to build my own index of the entire project inside my head.

I needed to know where a request entered the system, which components it passed through, where state was modified, who generated a particular parameter, who called a particular method, which path was taken after a failure, and where the final answer was produced.

Only after building this index could I genuinely understand the project, rather than merely knowing that “it currently runs.”

Building that mental index was far more difficult than asking an AI to search through files.

But I deserved that pain.

All the design, reading, and organizational work that I had skipped through vibe coding eventually returned in another form.

AI had helped me generate the code earlier, but it had not taken responsibility for understanding the code on my behalf.

Technical debt does not disappear simply because the code was written by AI.

On the contrary, because AI can generate code more quickly, technical debt can accumulate more quickly as well.

Codex and Defensive Code

While reading the code, I noticed another clear characteristic of Codex-generated implementations: they contained an enormous amount of defensive code.

At almost every stage, Codex would repeatedly verify whether data existed, whether its type was correct, whether fields matched expectations, whether parameters were empty, and whether the current object had a particular attribute.

When viewed in isolation, these checks usually made sense.

Codex did not naturally trust that validation had already been performed upstream, nor did it proactively rely on assumptions that had already been established elsewhere in the call context.

From its perspective, checking again inside the current function was the safest implementation.

This is a highly reliable code-generation strategy, but not necessarily a good long-term engineering strategy.

When the same parameter is repeatedly checked in the Router, Planner, Validator, Executor, and Answer stages, the code becomes extremely verbose.

The actual business logic is surrounded by layers of protective code, making it difficult for developers to distinguish between checks that define genuine system boundaries and checks that are merely redundant.

During the reading and refactoring process, I therefore began asking Codex to reduce repetitive validation based on the established context.

Data that has already passed validation through a Pydantic model at a stable boundary does not need every internal function to revalidate every field.

Similarly, tool dependencies already guaranteed by the Tool Graph Validator should not be reimplemented as incomplete validation logic inside every execution function.

More defensive code is not always better.

What matters is knowing where the system needs to defend itself.

External input, model output, API responses, persisted data, and component boundaries require strict validation.

Inside those boundaries, the code should be able to trust the contracts that have already been established.

Otherwise, so-called defensive programming eventually becomes a large volume of noise that no one can confidently determine is safe to remove.

Restructuring the Project for Human Readers

After encountering these oversized files, I began directing Codex to split the code apart.

However, mechanically dividing one large file into several smaller files was not enough. If smaller files caused the call relationships to become more scattered, they could make the system equally difficult to understand.

I therefore required detailed comments at critical locations: which component called the current method, which file the method called next, where its inputs came from, and which component received its outputs.

These comments were not written for the AI.

AI could search symbols and call relationships directly. I needed this information to build an overall mental model quickly.

As the restructuring continued, the project gradually developed clearer directory boundaries:

app/nlu became responsible for natural-language understanding and contextual resolution.

app/agents became responsible for planning, execution, reflection, and answer generation.

app/service_runtime became responsible for Redis, PostgreSQL, and runtime storage.

app/api_runtime became responsible for the API knowledge base and internal API calls.

app/web_evidence became responsible for external evidence.

app/services.py retained only HTTP adaptation and dependency assembly.

Stable structures also gradually moved away from anonymous dictionaries and toward explicit Pydantic models.

Model outputs, API call results, session context, long-term memory, and final-answer inputs all began to have their own typed boundaries.

At that point, I finally understood that modularity is not about making the directory structure look tidy.

The value of modularity is that it reduces the amount of information a human must load into their mind at the same time in order to understand the system.

A file should allow a developer to answer the following questions:

What is it responsible for?

What is it not responsible for?

Which inputs does it trust?

What outputs does it promise to downstream components?

What I Ultimately Learned

This project significantly changed how I understand local models, RAG, agents, and AI-assisted programming.

I no longer believe that local deployment automatically means autonomous and controllable AI.

Local deployment only gives you control over the runtime environment. Model capability, knowledge quality, and reasoning performance still determine the final user experience.

I also no longer believe that RAG simply means placing search results into a prompt.

Effective retrieval must consider information density, context budgets, task stages, and factual boundaries.

I no longer allow vector search to determine the order of tool calls.

Relevant knowledge can be retrieved, but strict API dependencies must be controlled through structured plans and deterministic validators.

I no longer believe that adding more models can solve system-level problems.

Model fallback can improve availability, but it cannot repair an incorrect division of responsibilities.

I also no longer treat “the code runs” as meaning that development is complete.

Code must be understandable, verifiable, testable, and modifiable. Otherwise, it is merely a collection of implementations that happen to work at the current moment.

Most importantly, I began to understand vibe coding differently.

Vibe coding is not without value.

It allowed me to explore a large number of ideas in a short period of time, build prototypes quickly, validate APIs, integrate models, and cover a range of functionality that would have taken a single developer much longer to implement manually.

The problem is that the speed borrowed through vibe coding must eventually be repaid by the developer.

You can temporarily avoid reading the code. You can temporarily ignore the fact that the files are becoming larger and larger. You can allow AI to keep patching failed tests.

But once the project becomes real, every structural problem that was ignored will return.

AI can help me generate code, search code, reverse-engineer websites, add tests, and perform refactoring.

But it cannot replace my responsibility to understand the system.

Architectural boundaries, sources of truth, runtime rules, and final trade-offs must still be determined by me.

Conclusion

In the end, the project did satisfy my requirements.

However, compared with the final achievement of building a sports scores agent, the more valuable outcome was experiencing firsthand how a project could evolve from a local-model experiment, a simple chatbot, FAISS retrieval, an API knowledge base, a keyword-based decision flow, and multi-model fallback into a controlled, testable, and explainable agentic workflow.

Many of the early implementations do not look elegant in retrospect. Some of the designs were wrong from the very beginning.

But I do not believe the process was entirely wasted.

If I had not personally encountered the problem of rules data consuming the entire context window, I would never have genuinely understood context engineering.

If I had not experienced vector search selecting the wrong API, I would not have understood the difference between knowledge retrieval and tool planning.

If I had not maintained a keyword-based decision flow, I would not have understood why natural-language intent cannot be exhaustively represented through rules.

If I had not faced files containing tens of thousands of lines generated by Codex, I would not have truly appreciated human readability and module boundaries.

This project was not completed in a single pass according to one correct architecture diagram.

It only gradually became what it is today after I repeatedly discovered that “this approach does not work.”

And that may be the most honest summary of my entire development experience:

AI can dramatically accelerate trial and error, but it cannot allow developers to skip growth. Every piece of understanding temporarily sacrificed for speed must eventually be recovered through their own effort.

Deep Dive: Using Redis Hash Tags to Reduce Tail Latency in Redis Cluster Batch Reads

Truman — Mon, 12 Jan 2026 10:51:09 +0000

In Redis Open Source Cluster, the keyspace is partitioned into 16,384 hash slots. Each slot is served by exactly one node at any point in time, and the cluster achieves high availability through master/replica replication.
Slots are computed using CRC16 and modulo 16,384.
A key design goal of Cluster is no proxies and no server-side merging: clients are redirected (e.g., via MOVED / ASK) to the correct node and execute commands there.

This architecture often leads to a common production symptom: batch reads that are fast on a standalone Redis (e.g., MGET / pipelining) show much higher P99 jitter and occasional timeouts on Cluster.

1. Root Cause: Cross-slot fan-out amplifies tail latency

Redis Cluster has a clear boundary for “complex multi-key operations”: they are supported only when all involved keys hash to the same slot; otherwise, multi-key capabilities are not available.

Therefore, if a set of logically related keys is spread across multiple slots:

Variadic multi-key commands such as MGET/MSET are constrained (a common symptom is a CROSSSLOT error).
Even if MGET is replaced by multiple GETs or pipelining, the workload still becomes multi-node requests + client-side aggregation (scatter-gather), and overall latency is dominated by the slowest node.

2. Core Mechanism: Hash Tags reduce “M nodes” to “1 node”

Cluster provides hash tags to enable data affinity: force a related group of keys onto the same slot, thereby enabling multi-key operations and significantly reducing cross-node fan-out.

Hash tag rules (strictly per cluster-spec)

Hash tag processing is enabled only when all of the following are true:

The key contains {
There is a } to the right of that {
There is at least one character between the first { and the first } to its right

When valid, Cluster computes the CRC16 only on the substring inside that first valid {...} to determine the slot.

Empty {} is not a valid hash tag: for example, foo{}{bar} hashes the entire key, not the empty string.

3. Where the benefit comes from: less fan-out, not “faster Redis execution”

Example (Favorites scenario):

Before: favorites:123:news001 … favorites:123:news010
Different full key strings → likely different slots → MGET may be disallowed or suffer high jitter.
After: favorites:{...}:news001 … favorites:{...}:news010
Same tag → same slot → same node → collapse “multi-node concurrent requests + merge” into “single-node one round trip”.

4. Related Issues: practical boundaries and common misconceptions

1) If `{...}` contents are the same across services, will values collide?

No.
Hash tags only influence which slot/node a key maps to; they do not change key uniqueness. Overwrites happen only when the entire key string is identical.

The real risk is missing namespaces between domains/services, which can cause full-key collisions—not shared tag values.

2) Must the tag contain only a `userId`?

From a correctness standpoint, using only userId is fine.
From a load and “affinity boundary” standpoint, the tag should express which keys truly must be colocated.

If multiple domains share {userId}, the same user’s traffic across domains tends to concentrate on the same slot (no value confusion, but potential hot-spot stacking). A safer pattern is a composite tag that constrains affinity within a domain:

favorites:{fav:123}:news001
orders:{ord:123}:order8899

This preserves multi-key capability within the domain while reducing unnecessary cross-domain slot binding.

3) Even with hash tags, mixing different tags still fails

For example, a single MGET mixing {u1} and {u2} is still cross-slot. Cluster’s constraint remains “same slot”; hash tags only make “same slot” controllable by design.

5. Key Naming and Tag Design Guidelines (production-ready conventions)

This section can be used directly as an internal engineering guideline.

1) Namespacing (mandatory)

Prevent collisions across services/modules.

Recommended prefix structure:

<env>:<service>:<module>:...

Examples:

prod:news:favorites:{fav:123}:news:001
test:uc:session:{sess:uid123}:token

2) Tag placement and content (core)

Recommended structure:

<prefix>:{<affinity>}:<suffix>

Where <affinity> defines the “affinity boundary”. Prefer domain + primary dimension:

{fav:<userId>}: favorites grouped by user
{ord:<orderId>}: orders grouped by order
{cart:<userId>}: cart grouped by user

This yields:

Stable same-slot behavior within the domain (enabling multi-key / reducing fan-out)
Avoids unnecessary cross-domain binding even if the dimension is the same (e.g., same userId)

Note: Cluster uses only the first valid {...} for hashing; empty {} is invalid and the whole key is hashed instead.

3) Fields and hierarchy (readability + extensibility)

Use a consistent delimiter (commonly :) and make object types/fields explicit:

prod:news:favorites:{fav:123}:news:001 (object: news, id: 001)
prod:news:favorites:{fav:123}:meta (aggregate metadata)
prod:uc:profile:{uc:123}:base (user base profile)

4) Versioning (recommended)

Add versions to handle value schema upgrades safely:

prod:news:favorites:v1:{fav:123}:news:001
prod:news:favorites:v2:{fav:123}:news:001

5) Skew & hot keys (must evaluate)

Hash tags intentionally pin a group of keys to one slot. If the tag is too coarse, it can create skew. The cluster spec notes that hash tags exist to enable multi-key operations, and forcing too much into one slot leads to imbalance.

Avoid:

Global tags like {common} or {config}
Using a single tag to aggregate site-wide data

深度解析：利用 Redis Hash Tag 解决 Redis Cluster 下批量读取的延迟抖动

Truman — Mon, 12 Jan 2026 10:47:44 +0000

在 Redis Cluster 中，keyspace 会被切分到 16384 个 hash slot，每个 slot 在任意时刻只由一个节点负责；集群通过 master/replica 复制实现高可用。(Redis)
slot 的计算基于 CRC16，并对 16384 取模。(Redis)
同时，Cluster 设计目标之一就是 无代理（no proxies）、不做服务端 merge 操作，客户端通过重定向（如 MOVED / ASK）连接到正确节点执行。(Redis)

这套设计带来一个典型现象：单机里很快的批量读（MGET / pipeline）到 Cluster 上，P99 抖动明显变大，甚至超时。

一、问题根因：跨 slot 的 fan-out 放大尾延迟

Redis Cluster 对“复杂多 key 操作”有明确边界：只有当涉及的 keys 都 hash 到同一 slot 时才支持；否则 multi-key 能力会不可用。(Redis)

因此，业务上一批相关 key 如果分散到多个 slot：

MGET/MSET 这类 variadic multi-key 命令会受限（常见表现是 CROSSSLOT）。
即便不使用 MGET，而是改成多次 GET 或 pipeline，本质上仍会变成 多节点请求 + 合并结果（scatter-gather），整体耗时由最慢节点拖累。

二、核心技术：Hash Tag 的作用是把 M 个节点收敛到 1 个节点

Cluster 通过 hash tag 提供“数据亲和性”：强制一组相关 key 落在同一 slot，从而允许 multi-key 操作并显著降低跨节点 fan-out。(Redis)

Hash Tag 规则（严格按 cluster-spec）

当且仅当满足以下条件时启用 {...}：(Redis)

key 包含 {
{ 右侧存在 }
第一对 { 与其右侧第一个 } 之间至少有 1 个字符

满足时：只对这段子串做 CRC16 以计算 slot。(Redis)

空 {} 不生效：例如 foo{}{bar} 会对整串 key 正常哈希，而不是对空串哈希。(Redis)

三、收益来自哪里：减少跨节点 fan-out，而不是“Redis 执行更快”

示例（收藏场景）：

优化前：favorites:123:news001 … favorites:123:news010
key 全名不同 → slot 大概率不同 → 原生 MGET 会因跨 slot 报错，而客户端模拟的批量操作会因抖动导致延迟剧增。
优化后：favorites:{...}:news001 … favorites:{...}:news010
tag 相同 → 同一 slot → 同一节点 → 从“多节点并发 + 合并”收敛为“单节点一次返回”。

四、相关问题（工程边界与常见误解）

1）`{...}` 内容相同会不会“值乱/冲突”？

不会。
Hash Tag 只影响 落在哪个 slot/节点，不会影响 key 的唯一性；是否覆盖只取决于 完整 key 字符串是否相同。

真正需要防的是：不同业务/不同服务没做命名空间隔离导致完整 key 撞名，而不是 tag 重复。

2）Tag 是否只能放一个 `userId`？

从正确性角度：只放 userId 完全可行。(Redis)
从负载与“亲和边界”角度：tag 的内容应表达“哪些 key 必须放在一起”。

如果多个业务域都用 {userId}，同一用户的跨业务访问会更倾向集中到同一 slot（不乱值，但可能叠加热点）。更稳的做法是使用 复合 tag 限定亲和边界，例如：

favorites:{fav:123}:news001
orders:{ord:123}:order8899

这样仍可保证“服务内/业务内” multi-key 能力，同时减少“跨业务不必要绑死在同一 slot”的概率。

3）即便用了 Hash Tag，混用不同 Tag 仍会失败

例如一次 MGET 混合 {u1} 与 {u2}，本质上仍是跨 slot，multi-key 约束依然不满足。Redis Cluster 的 multi-key 能力边界是“同 slot”，hash tag 只是让“同 slot”变得可控。(Redis)

五、Key 命名规范与 Tag 设计准则（落地规范）

下面这段可以直接作为团队规范的“统一模板”。

1）命名空间（必须）

保证不同服务/不同模块不会撞 key。

推荐前缀结构：

<env>:<service>:<module>:...

示例：

prod:news:favorites:{fav:123}:news:001
test:uc:session:{sess:uid123}:token

2）Tag 放置与内容（核心）

推荐结构：

<prefix>:{<affinity>}:<suffix>

其中 <affinity> 用于定义“亲和边界”，建议采用 业务域 + 主维度：

{fav:<userId>}：收藏业务按用户聚合
{ord:<orderId>}：订单业务按订单聚合
{cart:<userId>}：购物车按用户聚合

这样做的效果是：

同业务内可 multi-key（同 slot）
跨业务不会因为同一个 userId 产生不必要的 slot 绑定

注意：Cluster 只会使用第一对有效 {...} 参与哈希；空 {} 会被当作无效，整串 key 哈希。(Redis)

3）字段与层级（可读性 + 可扩展）

建议用固定分隔符（常用 :），并把“对象类型/字段”显式写出来：

prod:news:favorites:{fav:123}:news:001（对象：news，id：001）
prod:news:favorites:{fav:123}:meta（聚合元信息）
prod:uc:profile:{uc:123}:base（用户基础资料）

4）版本号（建议）

当 value 结构可能升级时，加版本避免回滚/灰度混乱：

prod:news:favorites:v1:{fav:123}:news:001
prod:news:favorites:v2:{fav:123}:news:001

5）热点与倾斜（必须评估）

Hash Tag 会把一组 key 固定到同一 slot；tag 粒度过粗会造成倾斜。Cluster-spec 也明确 hash tag 的用途是为了 multi-key，同样需要避免把大量不相关 key 强行塞到一个 slot。(Redis)

禁止示例：

{common}、{config} 这类全局 tag
用一个 tag 聚合“全站数据”

Gracefully Integrating Sentry-Go Middleware into Fiber v3 Projects

Truman — Tue, 15 Jul 2025 03:49:02 +0000

Gracefully Integrating Sentry-Go Middleware into Fiber v3 Projects

This article explains how to integrate error monitoring functionality using Sentry-Go v0.34.1 in projects built with Go 1.24.3 and Fiber v3. We'll demonstrate how to adapt and rewrite the sentryfiber middleware to support Fiber v3 and enhance flexibility with configurable path skipping (SkipPaths).

Technology Stack

Go Version: 1.24.3
Fiber Version: v3
Sentry-Go Version: 0.34.1

Currently, the official sentryfiber middleware supports only Fiber v2. Given our project's upgrade to Fiber v3, middleware adaptation and rewriting are necessary.

Overview of Custom Middleware Features

The rewritten middleware introduces these key enhancements:

Fiber v3 Compatibility: Ensures compatibility with all Fiber v3 APIs.
SkipPaths Configuration: Allows specific request paths to be ignored, reducing redundant or unnecessary logging.
Customizable Options: Includes common configurations such as Repanic, WaitForDelivery, and Timeout, providing a flexible integration experience.

Implementation

Here is the key implementation code:

type Options struct {
    Repanic         bool
    WaitForDelivery bool
    Timeout         time.Duration
    SkipPaths       []string
}

func New(opts Options) fiber.Handler {
    if opts.Timeout == 0 {
        opts.Timeout = 2 * time.Second
    }
    skip := opts.SkipPaths

    return func(c fiber.Ctx) error {
        if len(skip) > 0 {
            path := c.Path()
            for

在 Fiber v3 项目中优雅地集成 Sentry-Go 中间件

Truman — Tue, 15 Jul 2025 03:47:52 +0000

在 Fiber v3 项目中优雅地集成 Sentry-Go 中间件

本文将介绍如何在使用 Go 1.24.3 和 Fiber v3 的项目中，使用 Sentry-Go v0.34.1 集成错误监控功能，并通过自定义 sentryfiber 中间件支持 Fiber v3，增加了灵活的路径跳过配置 (SkipPaths)。

技术选型

Go 版本：1.24.3
Fiber 版本：v3
Sentry-Go 版本：0.34.1

Sentry 官方的 sentryfiber 中间件目前只支持 Fiber v2，考虑到我们项目已经升级至 Fiber v3，因此需要进行适配并重写相关中间件。

自定义中间件功能概述

我们重写的中间件新增了以下重要功能：

兼容 Fiber v3：确保所有 API 与 Fiber v3 兼容。
SkipPaths 配置：通过路径配置来忽略某些请求，避免冗余或不必要的日志上报。
自定义选项：支持如 Repanic、WaitForDelivery 和 Timeout 等常用配置，提供灵活的集成体验。

实现方式

以下为关键实现代码：

type Options struct {
    Repanic         bool
    WaitForDelivery bool
    Timeout         time.Duration
    SkipPaths       []string
}

func New(opts Options) fiber.Handler {
    if opts.Timeout == 0 {
        opts.Timeout = 2 * time.Second
    }
    skip := opts.SkipPaths

    return func(c fiber.Ctx) error {
        if len(skip) > 0 {
            path := c.Path()
            for _, s := range skip {
                if path == s || strings.HasPrefix(path, s) {
                    return c.Next()
                }
            }
        }

        hub := sentry.CurrentHub().Clone()
        ctxWithHub := sentry.SetHubOnContext(context.Background(), hub)
        c.SetContext(ctxWithHub)

        hub.Scope().SetRequest(toHTTPRequest(c))
        traceHeader := string(c.Get(sentry.SentryTraceHeader))
        baggageHeader := string(c.Get(sentry.SentryBaggageHeader))
        tx := sentry.StartTransaction(
            ctxWithHub,
            string(c.Method())+" "+c.Path(),
            sentry.WithOpName("http.server"),
            sentry.ContinueFromHeaders(traceHeader, baggageHeader),
        )
        hub.Scope().SetSpan(tx)
        defer tx.Finish()

        defer func() {
            if r := recover(); r != nil {
                hub.Recover(r)
                hub.Flush(opts.Timeout)
                if opts.Repanic {
                    panic(r)
                }
            }
        }()

        if err := c.Next(); err != nil {
            hub.CaptureException(err)
            return err
        }

        if opts.WaitForDelivery {
            hub.Flush(opts.Timeout)
        }
        return nil
    }
}

SkipPaths 的应用场景

通过配置 SkipPaths 可以有效避免特定路由请求的监控，比如健康检查接口：

app.Use(middle.New(middle.Options{
    Repanic:         true,
    WaitForDelivery: false,
    Timeout:         3 * time.Second,
    SkipPaths:       []string{"/health", "/metrics"},
}))

小结

本文通过对官方 sentryfiber 中间件的重写，提供了 Fiber v3 项目使用 Sentry-Go 的最佳实践，并显著提升了日志监控的灵活性和准确性。

WebSocket 请求通过 CDN 返回 403 Forbidden 的问题排查与解决

Truman — Sat, 24 May 2025 05:31:59 +0000

WebSocket 请求通过 CDN 返回 403 Forbidden 的问题排查与解决（wstest.example.com）

在一次使用 Kubernetes + NGINX Ingress + WebSocket 服务部署中，配置了 CDN（Funnell）接入域名：

wss://wstest.example.com/push/ws?token=xxx

服务端为 Go 编写的 go-push 应用，监听端口 8081，用于处理 WebSocket 请求。

问题描述

客户端（如 Postman、浏览器）访问 WebSocket 接口时，始终收到：

403 Forbidden

响应头极其简洁，仅包含：

Date: ...
Content-Length: 0

后端服务日志中无任何请求记录，初步判断 请求未到达服务层。

排查过程

1. 检查 Ingress 配置

Kubernetes Ingress 使用以下配置：

annotations:
  nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
  nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
  nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
  nginx.ingress.kubernetes.io/rewrite-target: /$2
  nginx.ingress.kubernetes.io/configuration-snippet: |
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header X-Request-From gateway;

路径路由为：

- path: /push(/|$)(.*)
  pathType: ImplementationSpecific
  backend:
    service:
      name: go-push
      port:
        number: 8081

确认路由正确，路径 /push/ws 会被重写为 /ws 并转发给 go-push 服务。

2. 测试绕过 CDN，直连 Ingress

使用 curl 模拟 WebSocket 握手：

curl -i -N \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Key: test123==" \
  -H "Host: wstest.example.com" \
  http://<Ingress-IP>:<Port>/push/ws

返回 101 Switching Protocols，说明 Kubernetes 服务与 Ingress 正常，问题来自 CDN。

3. 确认 CDN（Funnell）拒绝连接

日志及响应特征表明：403 来自 CDN 层拦截而非应用或 Ingress 返回。主要怀疑：

WebSocket 未开启支持
请求路径未允许回源
请求头未透传

解决方案（Funnell 配置）

在 Funnell CDN 控制台中，对域名 wstest.example.com 做如下配置调整：

开启 WebSocket 支持

开启「WebSocket 协议支持」或「连接升级（Upgrade）」功能

配置回源路径白名单

添加 /push/.* 至允许回源路径
若默认只允许 /api 或 /static，未配置的新路径可能被直接 403 拦截

请求头透传设置

保证以下头部 不被 CDN 移除或篡改：

Header	正确值
`Connection`	`Upgrade`
`Upgrade`	`websocket`
`Sec-WebSocket-*`	保留原样

验证成功

完成配置后，再次通过 WebSocket 客户端访问：

wss://wstest.example.com/push/ws?token=xxx

得到正确响应：

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade

服务端日志打印握手成功，连接稳定。

总结

当 WebSocket 请求经由 CDN（如 Funnell）出现 403 Forbidden，常见原因不在服务本身，而是 CDN 安全策略或协议限制。请务必检查以下：

CDN 是否开启 WebSocket 支持
回源路径是否白名单允许
是否保留关键握手头部（如 Upgrade、Connection）
Ingress 是否配置正确的 proxy_set_header 和路径转发规则

处理得当后，可实现 CDN 加速 + WebSocket 稳定通信的双赢方案。

Troubleshooting and Resolving WebSocket 403 Forbidden Errors Through CDN

Truman — Sat, 24 May 2025 05:09:39 +0000

Troubleshooting and Resolving WebSocket 403 Forbidden Errors Through CDN (wstest.example.com)

）##

In a Kubernetes deployment using NGINX Ingress and a Go-based WebSocket service (go-push), we encountered a problem after routing traffic through a CDN provider (Funnell) using the following domain:

wss://wstest.example.com/push/ws?token=xxx

The backend service listens on port 8081 and is designed to handle WebSocket connections.

The Problem

When connecting to the WebSocket endpoint via a browser or tools like Postman, we consistently received:

403 Forbidden

With minimal response headers:：

Date: ...
Content-Length: 0

No requests were seen in the backend service logs, indicating the request never reached the application layer.

Investigation Steps

1. Check Ingress Configuration

The Kubernetes Ingress was configured with the following annotations:

annotations:
  nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
  nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
  nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
  nginx.ingress.kubernetes.io/rewrite-target: /$2
  nginx.ingress.kubernetes.io/configuration-snippet: |
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header X-Request-From gateway;

The routing rule was defined as:

- path: /push(/|$)(.*)
  pathType: ImplementationSpecific
  backend:
    service:
      name: go-push
      port:
        number: 8081

This configuration rewrites /push/ws to /ws and forwards it to the backend service.

2. Bypass the CDN to Verify Ingress Functionality

I used curl to simulate a WebSocket handshake request:

curl -i -N \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Key: test123==" \
  -H "Host: wstest.example.com" \
  http://<Ingress-IP>:<Port>/push/ws

I received a 101 Switching Protocols response, confirming that both Ingress and the backend were functioning correctly. The issue was clearly upstream—in the CDN layer.

3. Identify CDN as the Source of the 403

Based on the response and headers, we confirmed the 403 Forbidden was returned by the CDN (Funnell), not by Kubernetes or the backend service. Likely causes included:

WebSocket support not enabled
The /push/ws path not allowed in origin routing
Required WebSocket headers being stripped

Solution (Funnell CDN Configuration)

To resolve this issue, we made the following changes in the Funnell CDN management console for the domain wstest.example.com:

Enable WebSocket Support

Activate the WebSocket support or Connection Upgrade option in the CDN settings

Allow the WebSocket Path in Origin Routing

Explicitly allow paths like /push/.* in the origin path whitelist
If the default routing only allows /api or /static, custom paths will be blocked with 403

Preserve WebSocket Headers

Ensure the following headers are forwarded without modification:

Header	Expected Value
`Connection`	`Upgrade`
`Upgrade`	`websocket`
`Sec-WebSocket-*`	As-is

Successful Validation

After updating the CDN settings, WebSocket handshake requests to:

wss://wstest.example.com/push/ws?token=xxx

succeeded with a valid response:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade

And the backend logs confirmed the connection was received and processed correctly.

Conclusion

When WebSocket requests routed through a CDN return 403 Forbidden, the root cause is often not in the backend or Ingress but in the CDN's security and protocol handling rules.

To ensure WebSocket connections work correctly through a CDN:

Enable WebSocket protocol support
Allow the WebSocket path in origin routing
Preserve critical handshake headers (Upgrade, Connection)
Use proper rewrite and header forwarding rules in Ingress

With these in place, it's possible to enjoy both CDN performance and stable WebSocket connectivity.

Java 中 Redis 存储泛型对象 Map 的反序列化陷阱

Truman — Thu, 01 May 2025 07:47:19 +0000

背景介绍

本项目采用了双层缓存机制：

本地缓存使用 Caffeine 2.8.8
远程缓存使用 Redis
系统运行在 JDK 1.8

我们将一个包含多个字段的 Map<String, Object> 对象整体写入 Redis，其中 "list" 字段的值为一个 List<NewLiveMatchVO> 类型的对象列表：

写入 Redis 前：值是 Java 内存中的 List<NewLiveMatchVO>
从 Redis 反序列化后：值变成了 List<LinkedHashMap>，由于类型信息丢失，强制转换会抛出 ClassCastException

以下是相关缓存处理代码片段：

// redis序列化代码
@Slf4j
public class RedisObjectSerializer implements RedisSerializer<Object> {
    private static final byte[] EMPTY_BYTE_ARRAY = new byte[0];

    private final Converter<byte[], Object> deserializingConverter = new DeserializingConverter();
    @Override
    public byte[] serialize(Object obj) {
        // 这个时候没有要序列化的对象出现，所以返回的字节数组应该就是一个空数组
        if (obj == null) {
            return EMPTY_BYTE_ARRAY;
        }
        // 将对象变为字节数组
        byte[] data = JSON.toJSONBytes(obj, SerializerFeature.WriteClassName);
        return data;
    }

    @Override
    public Object deserialize(byte[] data) {
        // 此时没有对象的内容信息
        if (data == null || data.length == 0) {
            return null;
        }
        try {
            return JSON.parse(data, Feature.SupportAutoType);
        } catch (Exception e) {
            try {
                return this.deserializingConverter.convert(data);
            } catch (Exception e1) {
                log.info("无法反序列化redis数据：{}", new String(data, StandardCharsets.UTF_8), e);
            }
            throw e;
        }
    }
}

//数据上游，填充数据阶段，返回map
Cache<String, Map<String, Object>> cache = GenericCacheManager
    .getWriteExpireCacheByName("queryData", 200, 120);
String cacheKey = "cache-key";
String redisKey = "redisKey";
Map<String, Object> finalResult = cache.get(cacheKey, method -> {
    Map<String, Object> redisCache = (Map<String, Object>) redisTemplate.opsForValue().get(redisKey);
    if (null != redisCache) {
        return redisCache;
    }
    Map<String, Object> map = Maps.newHashMap();
    List<NewLiveMatchVO> list = queryListData();
    map.put("list", list);
    Map<Integer, List<NewTournamentVO>> tCollect = list.stream().collect(Collectors.groupingBy(NewTournamentVO::getSportId));
    map.put("tournament", tCollect);
    redisTemplate.opsForValue().set(redisKey, map, 150, TimeUnit.SECONDS);
    return map;
});

// 获取缓存数据，强制类型转换（潜在风险点）
List<NewLiveMatchVO> matchList = (List<NewLiveMatchVO>) finalResult.get("list");

// 后续处理
List<?> process = processResult(matchList);

private List processResult(List<NewLiveMatchVO> list) {
    List items = new ArrayList();
    if (CollectionUtils.isEmpty(list)) {
        return items;
    }

    list = list.stream().distinct().collect(Collectors.toList());

    // 在这里报错 java.lang.ClassCastException: java.util.ArrayList cannot be cast to com.qiutx.product.model.vo.NewLiveMatchVO
    list.forEach(e -> {
        List item = new ArrayList();
        items.add(item);
    });
    return items;
}

根本原因

Fastjson 在序列化整个 Map 的时候，如果 Map 内的值（例如 List）没有显式类型信息（@type），它就不会自动对 List 内的泛型对象添加类型。

所以即使你用了 SerializerFeature.WriteClassName，最终只记录了 Map 和 Map 的 key 是 "list"，value 是个 List，而 NewLiveMatchVO 的类型信息丢失了。

反序列化回来就是这个样子：

Map<String, Object> finalResult = redisTemplate.opsForValue().get("xxx");

// 此时 finalResult.get("list") 实际是 List<LinkedHashMap>（不是 List<NewLiveMatchVO>）

去强转 List 就会炸：

(List<NewLiveMatchVO>) finalResult.get("list"); // ❌ ClassCastException

处理办法有很多，比如：只存 JSON 字符串、使用泛型反序列化（不推荐 Map），使用专门的 VO 包装类

🧠 总结一句话：

当你把整个 Map 存入 Redis 并反序列化回来后，其中的泛型类型（如 List<NewLiveMatchVO>）会被擦除为 List<LinkedHashMap>，
这会导致 ClassCastException，必须在使用前重新手动转换。

Native Memory Tracking 参数解释

Truman — Tue, 19 Nov 2024 08:21:19 +0000

此片仅为NMT的输出信息解释，详细内容在这里

启动命令：
-Xms8G -Xmx8G -Xmn4G -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256M -XX:NativeMemoryTracking=summary -XX:+UseParallelGC

输出信息：

sh-4.4# jcmd 1 VM.native_memory summary
1:

Native Memory Tracking:

(Omitting categories weighting less than 1KB)

Total: reserved=9488872KB, committed=9037604KB
       malloc: 90692KB #530181
       mmap:   reserved=9398180KB, committed=8946912KB

-                 Java Heap (reserved=8388608KB, committed=8388608KB)
                            (mmap: reserved=8388608KB, committed=8388608KB)

-                     Class (reserved=216083KB, committed=19475KB)
                            (classes #24831)
                            (  instance classes #23390, array classes #1441)
                            (malloc=3091KB #81714)
                            (mmap: reserved=212992KB, committed=16384KB)
                            (  Metadata:   )
                            (    reserved=131072KB, committed=126208KB)
                            (    used=125492KB)
                            (    waste=716KB =0.57%)
                            (  Class space:)
                            (    reserved=212992KB, committed=16384KB)
                            (    used=15382KB)
                            (    waste=1002KB =6.12%)

-                    Thread (reserved=97958KB, committed=10654KB)
                            (thread #96)
                            (stack: reserved=97644KB, committed=10340KB)
                            (malloc=205KB #579)
                            (arena=109KB #188)

-                      Code (reserved=254760KB, committed=95700KB)
                            (malloc=7072KB #22671)
                            (mmap: reserved=247688KB, committed=88628KB)

-                        GC (reserved=310290KB, committed=310286KB)
                            (malloc=6542KB #99)
                            (mmap: reserved=303748KB, committed=303744KB)

-                  Compiler (reserved=821KB, committed=821KB)
                            (malloc=657KB #1513)
                            (arena=164KB #4)

-                  Internal (reserved=1608KB, committed=1608KB)
                            (malloc=1572KB #53569)
                            (mmap: reserved=36KB, committed=36KB)

-                     Other (reserved=16890KB, committed=16890KB)
                            (malloc=16890KB #52)

-                    Symbol (reserved=41301KB, committed=41301KB)
                            (malloc=36179KB #334650)
                            (arena=5122KB #1)

-    Native Memory Tracking (reserved=8483KB, committed=8483KB)
                            (malloc=199KB #3578)
                            (tracking overhead=8284KB)

-        Shared class space (reserved=16384KB, committed=12956KB, readonly=0KB)
                            (mmap: reserved=16384KB, committed=12956KB)

-               Arena Chunk (reserved=692KB, committed=692KB)
                            (malloc=692KB #373)

-                    Module (reserved=252KB, committed=252KB)
                            (malloc=252KB #4553)

-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

-           Synchronization (reserved=2592KB, committed=2592KB)
                            (malloc=2592KB #25509)

-            Serviceability (reserved=18KB, committed=18KB)
                            (malloc=18KB #36)

-                 Metaspace (reserved=132121KB, committed=127257KB)
                            (malloc=1049KB #1245)
                            (mmap: reserved=131072KB, committed=126208KB)

-      String Deduplication (reserved=1KB, committed=1KB)
                            (malloc=1KB #8)

-           Object Monitors (reserved=4KB, committed=4KB)
                            (malloc=4KB #19)

JVM Native Memory Tracking (NMT) 解读

从 jcmd VM.native_memory summary 输出中分析 JVM 各部分内存使用情况：

1. JVM 内存总览

总内存：
- reserved=9488872KB (~9GB)：JVM 已保留的虚拟内存。
- committed=9037604KB (~8.6GB)：JVM 已向操作系统申请并实际使用的物理内存。
分配方式：
- malloc=90692KB：通过 C 的 malloc 分配的内存。
- mmap=9398180KB：通过内存映射（mmap）分配的内存。

2. Java 堆（Java Heap）

堆内存使用：
- reserved=8388608KB (~8GB)：最大堆大小。
- committed=8388608KB (~8GB)：已使用堆内存。
- 堆内存占用了 JVM 的绝大部分内存。
特点：
- 堆内存完全被 mmap 分配。
- 堆内存未出现碎片或不足问题。

3. Class（类加载器内存）

总内存：
- reserved=216083KB (~211MB)：已保留。
- committed=19475KB (~19MB)：实际使用。
- 其中，malloc 使用了 3091KB，mmap 使用了 16384KB。
元空间（Metadata）：
- reserved=131072KB (~128MB)：最大元空间大小。
- committed=126208KB (~123MB)：已使用。
- used=125492KB (~122.5MB)：元空间中的实际数据。
- 元空间碎片率：0.57%。
Class Space（类空间）：
- reserved=212992KB (~208MB)。
- committed=16384KB (~16MB)。
- 使用碎片率为 6.12%。

4. 线程（Thread）

线程总内存：
- reserved=97958KB (~95.6MB)：保留的线程相关内存。
- committed=10654KB (~10.4MB)：实际使用的线程相关内存。
- 当前线程数：96。
栈内存（Stack）：
- reserved=97644KB (~95.3MB)：线程栈的总保留内存。
- committed=10340KB (~10.1MB)：已实际分配的栈内存。
- 每个线程的栈大小：
```
平均栈大小 = committed / thread count = 10340KB / 96 ≈ 107.7KB
```

5. GC（垃圾回收器）

GC 内存使用：
- reserved=310290KB (~303MB)。
- committed=310286KB (~303MB)。
分配方式：
- malloc=6542KB (~6.4MB)。
- mmap=303748KB (~296.7MB)。
GC 内存分布：
- 主要分布在 mmap 中，可能用于 Parallel GC 的内部数据结构，如标记和回收的任务队列。

6. Code（代码缓存）

代码缓存：
- reserved=254760KB (~248MB)。
- committed=95700KB (~93MB)。
特点：
- 代码缓存用于存储 JIT 编译后的方法字节码。
- 其中 malloc=7072KB 和 mmap=247688KB。

7. Metaspace（元空间）

总内存：
- reserved=132121KB (~129MB)。
- committed=127257KB (~124MB)。
- 实际使用：
- 元数据区使用 used=125492KB。
- 类加载空间使用 used=15382KB。

8. Thread 栈空间的最大使用

线程栈使用：
- 当前线程数：96。
- 每个线程的平均实际栈使用量约为 107.7KB。
- 系统为每个线程预留了约 1MB 的栈内存空间。

总结

Java 堆内存：堆内存占用 8GB，无碎片，运行正常。
线程栈使用：
- 每个线程的平均实际栈使用量约为 107.7KB。
- 系统为每个线程预留了约 1MB 的栈内存空间。
元空间（Metaspace）：
- 使用了 124MB，运行正常。
代码缓存（Code Cache）：
- JIT 编译代码占用约 93MB。

如果线程栈需要更多内存，但未超出 -Xss 限制，则分配成功；如果超出限制，会触发 StackOverflowError。当前设置下，线程运行正常，没有栈不足问题。

内存增加后命令：
-Xms10G -Xmx10G -Xmn4G -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256M -XX:NativeMemoryTracking=summary -XX:+UseParallelGC

Kubernetes 监控发现使用内存 11.86GB。
输出信息：

sh-4.4# jcmd 1 VM.native_memory summary
1:

Native Memory Tracking:

(Omitting categories weighting less than 1KB)

Total: reserved=11658171KB, committed=11214347KB
       malloc: 91023KB #533380
       mmap:   reserved=11567148KB, committed=11123324KB

-                 Java Heap (reserved=10485760KB, committed=10485760KB)
                            (mmap: reserved=10485760KB, committed=10485760KB)

-                     Class (reserved=216153KB, committed=19609KB)
                            (classes #24834)
                            (  instance classes #23386, array classes #1448)
                            (malloc=3161KB #83460)
                            (mmap: reserved=212992KB, committed=16448KB)
                            (  Metadata:   )
                            (    reserved=131072KB, committed=126656KB)
                            (    used=125974KB)
                            (    waste=682KB =0.54%)
                            (  Class space:)
                            (    reserved=212992KB, committed=16448KB)
                            (    used=15385KB)
                            (    waste=1063KB =6.46%)

-                    Thread (reserved=91770KB, committed=9978KB)
                            (thread #90)
                            (stack: reserved=91476KB, committed=9684KB)
                            (malloc=192KB #543)
                            (arena=102KB #176)

-                      Code (reserved=255553KB, committed=97913KB)
                            (malloc=7865KB #23483)
                            (mmap: reserved=247688KB, committed=90048KB)

-                        GC (reserved=388402KB, committed=388398KB)
                            (malloc=6670KB #99)
                            (mmap: reserved=381732KB, committed=381728KB)

-                  Compiler (reserved=739KB, committed=739KB)
                            (malloc=575KB #1481)
                            (arena=164KB #4)

-                  Internal (reserved=1612KB, committed=1612KB)
                            (malloc=1576KB #54081)
                            (mmap: reserved=36KB, committed=36KB)

-                     Other (reserved=16849KB, committed=16849KB)
                            (malloc=16849KB #48)

-                    Symbol (reserved=41308KB, committed=41308KB)
                            (malloc=36186KB #334762)
                            (arena=5122KB #1)

-    Native Memory Tracking (reserved=8535KB, committed=8535KB)
                            (malloc=201KB #3621)
                            (tracking overhead=8334KB)

-        Shared class space (reserved=16384KB, committed=12956KB, readonly=0KB)
                            (mmap: reserved=16384KB, committed=12956KB)

-               Arena Chunk (reserved=69KB, committed=69KB)
                            (malloc=69KB #336)

-                    Module (reserved=252KB, committed=252KB)
                            (malloc=252KB #4553)

-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

-           Synchronization (reserved=2597KB, committed=2597KB)
                            (malloc=2597KB #25558)

-            Serviceability (reserved=18KB, committed=18KB)
                            (malloc=18KB #36)

-                 Metaspace (reserved=132158KB, committed=127742KB)
                            (malloc=1086KB #1281)
                            (mmap: reserved=131072KB, committed=126656KB)

-      String Deduplication (reserved=1KB, committed=1KB)
                            (malloc=1KB #8)

-           Object Monitors (reserved=3KB, committed=3KB)
                            (malloc=3KB #17)

JVM 内存对比分析（去除堆内存增加后的差异）

通过两次 jcmd VM.native_memory summary 输出，去除新增的 2GB 堆内存后，分析其他内存占用部分的变化。

1. 总内存

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Reserved	`9488872` (~9.0GB)	`11658171` (~11.1GB)	`+2179299` (~2GB)
Committed	`9037604` (~8.6GB)	`11214347` (~10.7GB)	`+2176743` (~2GB)

分析：

增长的 2GB 是堆内存的直接增加。
去除堆内存后，其他部分的变化如下。

2. 堆内存（Heap Memory）

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Reserved	`8388608` (~8GB)	`10485760` (~10GB)	`+2097152` (~2GB)
Committed	`8388608` (~8GB)	`10485760` (~10GB)	`+2097152` (~2GB)

分析：

堆内存完全按照配置变化，增加了 2GB。
去除堆内存影响后，以下部分差异进一步分析。

3. 元空间和类加载器内存

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Reserved	`216083` (~211MB)	`216153` (~211MB)	`+70` (~0MB)
Committed	`19475` (~19MB)	`19609` (~19MB)	`+134` (~0MB)
Metadata Used	`125492` (~122MB)	`125974` (~123MB)	`+482` (~0MB)

分析：

元空间使用量增加了约 0.5MB，主要因类加载数量增加（从 24831 个增加到 24834 个）。
增幅很小，属于正常的运行时类加载行为。

4. 线程（Thread Memory）

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Reserved	`97958` (~95MB)	`91770` (~89MB)	`-6188` (~6MB)
Committed	`10654` (~10MB)	`9978` (~9.7MB)	`-676` (~0.6MB)
线程数	`96`	`90`	`-6`

分析：

线程数量从 96 减少到 90，线程栈内存减少约 6MB。
单个线程栈平均大小保持不变，约为 107KB。

5. GC 内存（Garbage Collector Memory）

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Reserved	`310290` (~303MB)	`388402` (~379MB)	`+78112` (~76MB)
Committed	`310286` (~303MB)	`388398` (~379MB)	`+78112` (~76MB)

分析：

GC 内存增加了 76MB，可能是更大的堆导致 GC 数据结构（如标记表和任务队列）扩展。
属于垃圾回收器调整后的正常行为。

6. 代码缓存（Code Cache）

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Reserved	`254760` (~248MB)	`255553` (~249MB)	`+793` (~0.7MB)
Committed	`95700` (~93MB)	`97913` (~96MB)	`+2213` (~2MB)

分析：

代码缓存增加了约 2MB，与 JIT 编译的新代码有关。
增幅较小，对整体内存影响可以忽略。

7. Native Memory Tracking (NMT)

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Reserved	`8483` (~8MB)	`8535` (~8MB)	`+52` (~0MB)
Committed	`8483` (~8MB)	`8535` (~8MB)	`+52` (~0MB)

分析：

Native Memory Tracking 使用量略有增加，但变化非常小，仅为 52KB。

8. 其他部分

类别	第一次 (单位: KB)	第二次 (单位: KB)	差值 (单位: KB)
Other	`16890` (~16MB)	`16849` (~16MB)	`-41` (~0MB)
Symbol	`41301` (~40MB)	`41308` (~40MB)	`+7` (~0MB)
Metaspace	`132121` (~129MB)	`132158` (~129MB)	`+37` (~0MB)

分析：

这些部分的变化非常小，可以忽略。

整体对比总结

去除新增的堆内存后，其余部分内存的变化情况如下：

GC 内存：
- 增加了约 76MB，主要与更大的堆导致 GC 数据结构扩展有关。
线程内存：
- 减少了约 6MB，线程数量减少了 6 个。
元空间和类加载器：
- 增加了约 0.5MB，与加载更多类（增加 4 个类）有关。
代码缓存：
- 增加了约 2MB，与 JIT 编译的新代码有关。
Native Memory Tracking 和其他部分：
- 增加或减少幅度极小，影响可以忽略。

关键结论

主要变化来源：
- GC 内存 是增长的主要来源（+76MB）。
- 线程栈内存 减少了约 6MB。
总体变化幅度较小：
- 去除堆内存的增加后，整体的额外内存变化约为 +70MB，属于正常的运行时动态调整。
优化方向：
- GC 内存优化：
  - 如果需要降低 GC 开销，可以使用 G1GC 或调整 GC 的内存分配策略。
- 线程优化：
  - 避免多余线程的创建，进一步优化线程栈的内存分配。

总的来说，堆外内存的变化幅度很小，对应用整体内存影响有限。

JVM内存使用超出堆内存限制的问题排查

Truman — Tue, 19 Nov 2024 08:14:00 +0000

启动命令

-Xms8G -Xmx8G -Xmn4G -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256M -XX:+UseParallelGC

Kubernetes监控中的内存使用情况

根据启动命令的配置和 Kubernetes 显示的内存占用，出现了 JVM 内存（堆和非堆内存）与实际进程内存占用不一致的现象。

JVM 配置

堆内存

最小堆大小（-Xms）：8GB
最大堆大小（-Xmx）：8GB
年轻代大小（-Xmn）：4GB （年轻代是堆的一部分）

元空间（Metaspace）

初始元空间大小（-XX:MetaspaceSize）：256MB
最大元空间大小（-XX:MaxMetaspaceSize）：256MB

垃圾回收器

GC 类型：Parallel GC （吞吐量优先，但会占用更多内存来优化 GC 性能）

Kubernetes 监控显示

应用占用 9.4GB 内存，超出配置的 8GB 堆大小。

内存占用组成

JVM 应用的实际内存占用不仅包括堆内存，还包括以下部分：

1. 堆内存（Heap Memory）

定义为 -Xms 和 -Xmx 的 8GB 堆内存。
组成部分：
- 年轻代：配置为 -Xmn=4G。
- 老年代：存放从年轻代晋升的长生命周期对象。

2. 非堆内存（Non-Heap Memory）

元空间（Metaspace）：配置了最大 256MB (-XX:MaxMetaspaceSize=256MB)，但通常会稍微超过此值。
代码缓存区（Code Cache）：存储 JIT 编译后的代码，大小随 JIT 编译器的使用而增长。
线程栈内存（Thread Stack Memory）：
- 每个线程分配的栈内存由 -Xss 配置（默认 1MB）。
- 线程数量较多时，线程栈内存可能占用大量内存。

3. 堆外内存（Direct Memory）

使用 ByteBuffer.allocateDirect 或 NIO 时分配的直接内存。
默认情况下，直接内存大小与最大堆大小（-Xmx）相同，可通过 -XX:MaxDirectMemorySize 限制。

4. GC 线程和内部内存

GC 线程： Parallel GC 垃圾回收线程会占用本地内存和线程栈空间。
工作缓冲区： Parallel GC 会分配额外的工作缓冲区，用于垃圾回收操作，占用额外内存。

5. C 库或第三方库的内存

JNI 调用的本地库：通过 JNI 调用的库（如 Netty、OpenSSL 等）可能分配堆外内存。
这些内存由本地代码管理，不受 JVM 的直接控制。

可能原因分析

根据当前情况，以下是造成 Kubernetes 显示内存占用高于 JVM 配置的主要原因：

1. 线程栈内存消耗

默认线程栈大小：1MB（通过 -Xss 配置）。
影响：
- 如果应用使用大量线程（如处理高并发请求的线程池），线程栈内存会显著增加内存占用。
- 假设线程数量为 1000，线程栈内存占用约为： 1MB * 1000 = 1GB

2. 堆外内存（Direct Memory）

堆外内存的分配：
- 默认与最大堆大小一致（8GB）。
- 通常由 NIO 或框架（如 Netty）分配。
影响：
- 操作系统可能为堆外内存保留虚拟地址空间，即使实际使用未达到上限。

3. Parallel GC 的额外内存需求

GC 内存使用：
- Parallel GC 的多线程操作会分配额外内存用于内部数据结构（如标记、复制和整理）。
相关配置：
- GC 线程数与 CPU 核心数相关，通过 ParallelGCThreads 调整。
影响：
- 如果垃圾回收线程过多，内存占用会显著增加。

4. 元空间和代码缓存区增长

元空间的增长：
- 虽然配置了最大 256MB (-XX:MaxMetaspaceSize=256MB)，但 JVM 会动态分配更多内存用于类加载器和其他用途。
代码缓存区：
- 用于存储 JIT 编译后的代码，随着运行时间增加可能增长。
影响：
- 元空间和代码缓存区的增长可能导致非堆内存膨胀，进一步提升内存占用。

排查方法

以下方法可以帮助确认内存使用的具体来源：

1. 使用 Native Memory Tracking (NMT)

功能

NMT 可以精确追踪 JVM 内存分配，帮助识别内存的分布和使用情况。

启用 NMT

在 JVM 启动参数中添加以下配置：

-XX:NativeMemoryTracking=summary

查看内存分布

进入容器并运行：

jcmd 1 VM.native_memory summary
输出示例：

sh-4.4# jcmd 1 VM.native_memory summary
1:

Native Memory Tracking:

(Omitting categories weighting less than 1KB)

Total: reserved=9488872KB, committed=9037604KB
       malloc: 90692KB #530181
       mmap:   reserved=9398180KB, committed=8946912KB

-                 Java Heap (reserved=8388608KB, committed=8388608KB)
                            (mmap: reserved=8388608KB, committed=8388608KB)

-                     Class (reserved=216083KB, committed=19475KB)
                            (classes #24831)
                            (  instance classes #23390, array classes #1441)
                            (malloc=3091KB #81714)
                            (mmap: reserved=212992KB, committed=16384KB)
                            (  Metadata:   )
                            (    reserved=131072KB, committed=126208KB)
                            (    used=125492KB)
                            (    waste=716KB =0.57%)
                            (  Class space:)
                            (    reserved=212992KB, committed=16384KB)
                            (    used=15382KB)
                            (    waste=1002KB =6.12%)

-                    Thread (reserved=97958KB, committed=10654KB)
                            (thread #96)
                            (stack: reserved=97644KB, committed=10340KB)
                            (malloc=205KB #579)
                            (arena=109KB #188)

-                      Code (reserved=254760KB, committed=95700KB)
                            (malloc=7072KB #22671)
                            (mmap: reserved=247688KB, committed=88628KB)

-                        GC (reserved=310290KB, committed=310286KB)
                            (malloc=6542KB #99)
                            (mmap: reserved=303748KB, committed=303744KB)

-                  Compiler (reserved=821KB, committed=821KB)
                            (malloc=657KB #1513)
                            (arena=164KB #4)

-                  Internal (reserved=1608KB, committed=1608KB)
                            (malloc=1572KB #53569)
                            (mmap: reserved=36KB, committed=36KB)

-                     Other (reserved=16890KB, committed=16890KB)
                            (malloc=16890KB #52)

-                    Symbol (reserved=41301KB, committed=41301KB)
                            (malloc=36179KB #334650)
                            (arena=5122KB #1)

-    Native Memory Tracking (reserved=8483KB, committed=8483KB)
                            (malloc=199KB #3578)
                            (tracking overhead=8284KB)

-        Shared class space (reserved=16384KB, committed=12956KB, readonly=0KB)
                            (mmap: reserved=16384KB, committed=12956KB)

-               Arena Chunk (reserved=692KB, committed=692KB)
                            (malloc=692KB #373)

-                    Module (reserved=252KB, committed=252KB)
                            (malloc=252KB #4553)

-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

-           Synchronization (reserved=2592KB, committed=2592KB)
                            (malloc=2592KB #25509)

-            Serviceability (reserved=18KB, committed=18KB)
                            (malloc=18KB #36)

-                 Metaspace (reserved=132121KB, committed=127257KB)
                            (malloc=1049KB #1245)
                            (mmap: reserved=131072KB, committed=126208KB)

-      String Deduplication (reserved=1KB, committed=1KB)
                            (malloc=1KB #8)

-           Object Monitors (reserved=4KB, committed=4KB)
                            (malloc=4KB #19)

重点关注：

Thread：线程栈内存。
Direct：堆外内存。

2. 检查线程数量

进入 Pod 后，运行以下命令检查线程数：
ls /proc/<PID>/task | wc -l

输出示例

sh-4.4# ls /proc/1/task | wc -l
116

如果线程数量较多（数百甚至上千），需要检查线程池配置，避免过多线程占用栈内存。

3. 检查 Direct Memory

可以通过以下代码查看直接内存的使用情况：

System.out.println("Max Direct Memory: " + sun.misc.VM.maxDirectMemory() / (1024 * 1024) + " MB");

或者使用 jcmd 查看堆外内存分布。

4. 启用 GC 日志

在 JVM 启动参数中添加：
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/tmp/gc.log

分析垃圾回收日志，确认是否有过多的垃圾回收线程或堆外内存清理未及时释放。

优化建议

1. 减少线程栈内存

设置更小的线程栈大小（例如 512KB）：
-Xss512k

这可以显著减少线程栈内存占用。

2. 限制直接内存

设置直接内存的上限，例如 256MB：
-XX:MaxDirectMemorySize=256m

JVM内存使用超出堆内存限制的原因与优化方案

Truman — Tue, 19 Nov 2024 04:30:38 +0000

JVM内存使用超出堆内存限制的原因与优化方案

引言

在使用 Kubernetes 部署 JVM 应用时，经常会遇到这样的问题：即使在启动命令中明确设置了 JVM 的堆内存大小（例如 8GB），但通过 Kubernetes 监控却发现应用的实际内存使用超出了设置值（如 9.4GB）。更令人困惑的是，当进一步增加堆内存时，监控显示的内存使用量也会随之增长。这种现象可能导致内存超限、Pod 重启甚至应用不可用。

本文将深入分析 JVM 内存使用超出堆内存限制的原因，并提供针对性的优化方案，帮助开发者在实际项目中更高效地管理和优化内存使用。

JVM内存使用组成

JVM 的内存使用不仅仅包括堆内存，还涉及多个部分：

堆内存（Heap Memory）：
- 用于存储 Java 对象，大小由 -Xmx 和 -Xms 参数设置。
堆外内存（Off-Heap Memory）：
- 通过 DirectByteBuffer 或框架（如 Netty）分配的直接内存。
- 默认最大值与物理内存相关，可通过 -XX:MaxDirectMemorySize 调整。
本地方法区（Native Memory）：
- 包括线程栈（Thread Stack）、元空间（Metaspace）等。
- 线程栈大小由 -Xss 参数决定，线程数越多，消耗内存越大。
垃圾回收器的内存开销：
- 垃圾回收器（如 G1 GC）需要额外内存管理分代和回收任务。
容器内存开销：
- Kubernetes 监控容器的 RSS（Resident Set Size），包括 JVM 使用的内存和容器级别的内存开销。

常见问题分析

Kubernetes 监控的内存超出 JVM 堆内存限制：
- Kubernetes 监控的是容器内所有内存，而不仅仅是 JVM 堆内存。
- 堆外内存、线程栈等也会计入总内存使用。
增加堆内存后内存使用量同步增长：
- JVM 的非堆内存使用量可能与堆内存设置比例相关（如垃圾回收器的内存需求）。
- 应用本身可能存在内存泄漏或非堆内存过度使用的问题。

解决方案

1. 优化 JVM 内存设置

限制堆外内存：通过 -XX:MaxDirectMemorySize 设置堆外内存的最大值。
调整线程栈大小：通过 -Xss 参数减少线程栈内存（默认 1MB，可适当调低）。
合理设置堆大小：根据应用实际需求调整 -Xmx 和 -Xms，避免过度分配。

2. 加强内存监控

使用 JVM 自带工具：
- jstat：监控堆内存使用和垃圾回收信息。
- jmap：生成堆快照，分析内存分配。
引入第三方工具：
- VisualVM、JProfiler 等深入分析内存分配与线程使用情况。

3. 调整 Kubernetes 配置

配置内存限制：
- 在 Pod 的 resources 中设置合理的 memoryLimit 和 memoryRequest。
监控探针优化：
- 在 liveness 和 readiness 探针中加入内存使用监控，及时发现异常。

4. 优化代码逻辑

检查框架或库是否大量使用堆外内存（如 Netty、RocksDB 等）。
避免过多线程创建，优化线程池配置。
定期使用内存泄漏检测工具，如 Eclipse MAT。

总结

JVM 内存使用超出堆内存限制的现象常见于复杂部署环境中，其原因可能涉及堆外内存、本地方法区、线程开销等多个因素。通过合理优化 JVM 参数、监控内存使用和调整 Kubernetes 配置，可以有效降低内存使用并提高系统稳定性。

在实际项目中，建议结合工具深入分析内存分配，定位问题来源，以确保 JVM 应用在 Kubernetes 中的高效运行。

问题记录

Truman — Thu, 17 Oct 2024 06:39:44 +0000

Question 1
web 版本 1.0.1 不可以，1.0.2 就可以访问

Question 2
1/18 日用户模块用户添加和登陆出现问题，手机号没有查询到，导致重复添加的失败：「数据库主从同步出现问题」

Question 3
Nacos 2.2.3 服务注册不上的问题，最后通过查看 nacos 的源码，发现配置中的 namespace 已经不是旧版本的命名空间了，需要是命名空间的 ID。在 nacos 的控制台显示中，把 namespace 归为 Namespace ID, namespace showname 归为 namespaces。所以配置文件中的 namespace 要和 nacos 命名中对应的 Namespace ID 一致

Question 4
发现一个问题，当方法添加 @Cacheable 和@TimeToLive。缓存注解的时候，如果方法抛出 businessexception，之前的切面无法捕获并正常返回异常信息。
异常触发点：WarmReplayCacheInterceptorPostProcessor.class -> in.getMethod().invoke(advice, in.getArguments());
异常被捕获，此时的异常为 InvocationTargetException，包裹着 BusinessException。
异常中间流转的过程还没看，最后一次流转在 CglibAopProxy.class ->throw new UndeclaredThrowableException(ex);
到达项目中的异常统一处理方法 handleException，进入方法的异常为 UndeclaredThrowableException，在目前的方法处理中，是没有对 UndeclaredThrowableException 进行处理的，对此异常进行处理就行。
UndeclaredThrowableExceptione.getCause()这是 InvocationTargetException
UndeclaredThrowableExceptione.getCause().getCause()取出 BusinessException
然后按照 BusinessException 正常处理就行了

目前的问题是：1 没有加 else，返回 500；2:Throwable cause = e.getCause().getCause();
这个 cause 的值会随机出现两种样态
1:class com.qis.base.exception.BusinessException
2:class com.qis.base.exception.BusinessException message: 报错信息

在最新的测试中，不使用 instanceof 判断异常的类型，使用 cause.getClass().equals(BusinessException.class)进行判断，判断为 BusinessException 更加准确

Question 5
live_room_online_user 表，非查询操作非常频繁。执行清除直播间过期用户数据时，容易出现死锁问题。具体的操作是删除最后更新时间在 3 分钟前的数据。
第一次: 使用 FOR UPDATE 语句，这将为要删除的行获取排它锁，防止其他事务在删除过程中修改它们。
第二次: 分段删除，先查出主键数据，例如 ID 的值，再根据 ID 进行删除操作。如果删除的数据量太大，可以分段进行，从而降低死锁的可能性。
第三次，批量添加中使用 ON DUPLICATE KEY UPDATE，在遇到唯一键冲突时进行update操作
第四次，排查全部相关的DML，确保没有其他修改或者删除操作影响批量添加，事物propagation使用REQUIRE_NEW
第五次，使用insert ignore 替代 ON DUPLICATE KEY UPDATE，容忍个别数据未更新，避免锁的数据太多，容易死锁
第六次，不使用Transaction注解，改为手动控制；SQL执行操作还是使用MyBatis。批量更新的数组进行拆分，更新更小批量的数据。使用SqlSessionFactory进行更精确的手动控制，
关闭自动提交事物，每次小批量更新数据后手动提交事物，减少事物的长度，手动增加重试次数，第一次失败等待200ms，第二次失败等待400ms，第三次失败等待800ms，继续失败，丢弃数据

Question 6
Kubernetes 网关出现问题，经运维修复后，部分前端服务调用后端服务还是会出现 499 httpcode，导致前端服务无法正常提供。后端排查后所有服务无异常，可以使用 postman 调用，在 pod 的控制台中调用其他服务的测试接口，正常返回，可以确认后端服务无异常。解决办法是把前端有问题的服务重启，这样服务就恢复了。推测原因可能是网关问题修复后，Kubernetes 中的服务需要重启，在网关中重新注册，才能恢复正常。

Question 7
Kubernetes prometheus 容器挂掉，无法提供服务。原因是什么？集群进行压力测试，导致资源被占用，prometheus 容器无法获取到足够的资源，服务失败，只能运维人员手动重启

Question 8
查询当天足篮类型一级联赛的所有比赛，查询超时。原因是在 WHERE 语句中使用了 TO_DAYS 函数，导致索引失效，全表扫描。比赛时间本身是有索引的，所以将函数去除，改为传值比较，恢复索引。
并不是所有情况下在 WHERE 子句中使用函数都会导致索引失效。是否失效取决于函数的类型、函数的作用对象以及数据库引擎的优化能力。

会导致索引失效的情况对列进行转换或计算的函数：如果在 WHERE 子句中对索引列使用了改变其原始值的函数，那么通常会导致索引失效。例如： TO_DAYS(column)：将日期时间转换为天数。 UPPER(column)：将字符串转换为大写。 ABS(column)：求绝对值。这些函数都改变了列的原始值，数据库无法直接利用原始值上的索引，因此会导致索引失效，可能导致全表扫描。
不会导致索引失效的情况不改变列值的函数：有些函数不会改变列的实际值或可以被数据库优化器识别，从而不会导致索引失效。例如：范围查询：使用范围运算符（>、<、BETWEEN）通常不会导致索引失效函数作用在常量或表达式上如果函数只作用在常量或表达式上，而不是列上，索引仍然可以有效数据库优化器的特殊优化在一些数据库引擎中，某些函数或操作符经过优化后，仍然可以使用索引。例如在 MySQL 中，LIKE 'abc%' 可以利用索引，而 LIKE '%abc' 则不行。
数据库支持的函数索引支持基于函数结果的索引（称为函数索引或表达式索引）

Question 9
安卓安装包文件体积 100 多 MB，文件使用分片上传的方式传到华为云的 OBS 服务器。在分片上传的请求中，有些请求会出现 504 Gateway Time-out 。
在不使用 VPN 的情况下，请求链路从本地网络到达香港的华为云，从华为云到达项目的 kubernetes 环境，从 Kubernetes 集群进入，到达前端的管理后台服务，前端的管理后台服务在集群内调用后端的 backend 服务，backend 服务再调用 support 服务，在 support 服务中将文件上传到华为的 OBS 服务器。

第一次解决：推测是 backend 和 support 远程调用服务请求用时太长，导致请求中断连接。在项目通用配置中，设置 backend 和 support 远程调用服务的 FeignHttpClientProperties，将请求超时时间设置为 8 秒。需要设置 @primary ，不然会报错 KubernetesFeignClientConfiguration required a single bean, but 2 were found，一个是默认的(org.springframework.cloud.openfeign.support.FeignHttpClientProperties)，一个是自己定义的。

    @Bean
    @Primary
    @ConditionalOnExpression("'${spring.application.name:undefined}'=='backend' or '${spring.application.name:undefined}'=='support-rpc'")
    @ConfigurationProperties(prefix = "feign.httpclient")
    public FeignHttpClientProperties feignHttpClientProperties() {
        FeignHttpClientProperties properties = new FeignHttpClientProperties();
        properties.setConnectionTimeout(8000);  // Set connection timeout to 8000 ms
        return properties;
    }

第二次解决：在将配置发布到生产环境后，依旧会发生问题，说明问题没有真的解决，于是开启日志的逐行分析。在出现 504 请求后，推测分片上传的文件的 ID，根据文件 ID 查询后端的日志，将请求达到时间与上传成功时间记录，正常的请求和异常的请求分别记录三条。经过分析后，正常的请求用时一般在 2 秒以内，异常的请求在 62 到 63 秒之间。可以确定是异常的请求用时太长，客户端主动断开链接。为什么请求用时太长的问题，这个很难排查出一个确定的原因，可以推测是在香港华为云上的项目 kubernetes 环境与华为云 OBS 之间网络会偶尔出现一点问题，导致上传时间用时 62 到 63 秒。不过问题的直接原因是前端的管理后台服务中断了用时超长的请求，具体还是要前段同事去解决问题。

Question 10
项目优化的方式
API 请求分析，GC 信息记录，Kubernetes 服务实例的信息记录，比如内存使用记录和 CPU 使用记录
ZGC jdk 21.0.4: ZGC Major Collection (Proactive) 日志详解，ZGC 的两种主要的垃圾收集类型
ParallelGC jdk 1.8: ParallelGC 日志详解
FullGCListener: FullGCListener
SQL 耗时跟踪: SQL 耗时跟踪

Question 11
Kubernetes 中，流水线正常启动，但是容器无法正常启动，一闪而过 OOMKILLED，然后变成启动失败的重启状态。
原因是容器初始限制最多 2G 内存，但是 JAVA 启动命令请求 4G，年轻代设置 2G，内存请求达到限制门槛，但是还不够正常启动，于是发生 OOMKILLED，容器启动失败。

Question 12
线上服务报错，错误信息 java.lang.StackOverflowError ，已知设置的运行内存和 metaspace 空间大小都是充足的，查看报错日志，大量的阿里巴巴 fastjson 报错。 fastjson 版本为 1.2.60 ，最近的代码变动是使用 fastjson 中的 JSON.toJSONString 方法，推测是 JSON.toJSONString 方法的循环递归调用遇见未知问题，导致发生 java.lang.StackOverflowError，将代码变动回滚，重新发布，错误消除。

DEV Community: Truman

From the Fantasy of Local Models to a Truly Controllable Sports Agent

Starting a Chatbot as a Way to Make Use of the Hardware

The Second Collision with the API Knowledge Base and Vector Retrieval

Multi-Model Routing Did Not Solve the Architectural Problem

The Decision Flow and the Debt Created by Keyword Matching

Sports APIs Were Not Designed for LLMs

Deliberately Narrowing the Scope

From “Continuing to Patch” to Redesigning the System

Web Evidence Changed How I Understood “Internet Access”

AI Can Read One Hundred Thousand Lines of Code. Humans Cannot.

Codex and Defensive Code

Restructuring the Project for Human Readers

What I Ultimately Learned

Conclusion

Deep Dive: Using Redis Hash Tags to Reduce Tail Latency in Redis Cluster Batch Reads

1. Root Cause: Cross-slot fan-out amplifies tail latency

2. Core Mechanism: Hash Tags reduce “M nodes” to “1 node”

Hash tag rules (strictly per cluster-spec)

3. Where the benefit comes from: less fan-out, not “faster Redis execution”

4. Related Issues: practical boundaries and common misconceptions

1) If {...} contents are the same across services, will values collide?

2) Must the tag contain only a userId?

3) Even with hash tags, mixing different tags still fails

5. Key Naming and Tag Design Guidelines (production-ready conventions)

1) Namespacing (mandatory)

2) Tag placement and content (core)

3) Fields and hierarchy (readability + extensibility)

4) Versioning (recommended)

5) Skew & hot keys (must evaluate)

深度解析：利用 Redis Hash Tag 解决 Redis Cluster 下批量读取的延迟抖动

一、问题根因：跨 slot 的 fan-out 放大尾延迟

二、核心技术：Hash Tag 的作用是把 M 个节点收敛到 1 个节点

Hash Tag 规则（严格按 cluster-spec）

三、收益来自哪里：减少跨节点 fan-out，而不是“Redis 执行更快”

四、相关问题（工程边界与常见误解）

1）{...} 内容相同会不会“值乱/冲突”？

2）Tag 是否只能放一个 userId？

3）即便用了 Hash Tag，混用不同 Tag 仍会失败

五、Key 命名规范与 Tag 设计准则（落地规范）

1）命名空间（必须）

2）Tag 放置与内容（核心）

3）字段与层级（可读性 + 可扩展）

4）版本号（建议）

5）热点与倾斜（必须评估）

Gracefully Integrating Sentry-Go Middleware into Fiber v3 Projects

Gracefully Integrating Sentry-Go Middleware into Fiber v3 Projects

Technology Stack

Overview of Custom Middleware Features

Implementation

在 Fiber v3 项目中优雅地集成 Sentry-Go 中间件

在 Fiber v3 项目中优雅地集成 Sentry-Go 中间件

技术选型

自定义中间件功能概述

实现方式

SkipPaths 的应用场景

小结

WebSocket 请求通过 CDN 返回 403 Forbidden 的问题排查与解决

WebSocket 请求通过 CDN 返回 403 Forbidden 的问题排查与解决（wstest.example.com）

问题描述

排查过程

1. 检查 Ingress 配置

2. 测试绕过 CDN，直连 Ingress

3. 确认 CDN（Funnell）拒绝连接

解决方案（Funnell 配置）

开启 WebSocket 支持

配置回源路径白名单

请求头透传设置

验证成功

总结

Troubleshooting and Resolving WebSocket 403 Forbidden Errors Through CDN

Troubleshooting and Resolving WebSocket 403 Forbidden Errors Through CDN (wstest.example.com)

The Problem

Investigation Steps

1. Check Ingress Configuration

2. Bypass the CDN to Verify Ingress Functionality

3. Identify CDN as the Source of the 403

Solution (Funnell CDN Configuration)

Enable WebSocket Support

Allow the WebSocket Path in Origin Routing

1) If `{...}` contents are the same across services, will values collide?

2) Must the tag contain only a `userId`?

1）`{...}` 内容相同会不会“值乱/冲突”？

2）Tag 是否只能放一个 `userId`？