Jia

Posted on Jul 25

Add a Per-Tool Payload Size Guard Before Requests Reach Vector Engine

#llm

A shared Vector Engine setup often starts with a simple rule: point Dify, Cursor, and the Node.js backend at the same OpenAI-compatible API gateway, then keep the Base URL and model name consistent. That is necessary, but it does not cover payload size. Long prompts, pasted logs, oversized retrieved context, and generated tool output can all change the request before it reaches the LLM API provider layer.

This tutorial adds a small payload size guard in Node.js. It does not block useful work. It records message size, tool source, model name, and a clear failure category so model_not_found does not get confused with a request that should have been trimmed earlier.

Why size checks belong near the client

Dify may add retrieved knowledge chunks. Cursor may include code context. A Node.js service may attach user history or system instructions. If every tool sends traffic through Vector Engine, the provider route is shared, but the request shape is still tool-specific.

A per-tool guard gives you three benefits:

Signal	Engineering use
Tool name	Separates Dify, Cursor, and service traffic
Base URL	Confirms the client points to the intended Vector Engine endpoint
Model name	Keeps model routing visible
Payload bytes	Shows whether the request grew before the API call
Error class	Separates size policy from `model_not_found` and auth failures

A small Node.js wrapper

const BASE_URL = process.env.VECTOR_ENGINE_BASE_URL || "https://api.vectorengine.cn/v1";
const API_KEY = process.env.VECTOR_ENGINE_API_KEY;
const MODEL = process.env.VECTOR_ENGINE_MODEL;
const MAX_PAYLOAD_BYTES = Number(process.env.MAX_PAYLOAD_BYTES || 120000);

if (!API_KEY) throw new Error("VECTOR_ENGINE_API_KEY is required");
if (!MODEL) throw new Error("VECTOR_ENGINE_MODEL is required");

function byteLength(value) {
  return Buffer.byteLength(JSON.stringify(value), "utf8");
}

function classifyError(status, body) {
  const code = body?.error?.code;
  if (code === "model_not_found") return "model_route";
  if (status === 401 || status === 403) return "auth_or_key_scope";
  if (status === 404) return "base_url_or_route";
  if (status === 429) return "rate_limit";
  if (status >= 500) return "provider_or_upstream";
  return "request_or_unknown";
}

export async function callVectorEngine({ tool, messages }) {
  const payload = {
    model: MODEL,
    messages,
    temperature: 0.2,
  };

  const payloadBytes = byteLength(payload);
  const event = {
    tool,
    baseUrl: BASE_URL,
    model: MODEL,
    payloadBytes,
  };

  if (payloadBytes > MAX_PAYLOAD_BYTES) {
    return {
      ok: false,
      ...event,
      errorClass: "payload_size_policy",
      message: "Trim retrieved context, pasted logs, or conversation history before retrying.",
    };
  }

  const response = await fetch(`${BASE_URL.replace(/\/+$/, "")}/chat/completions`, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${API_KEY}`,
    },
    body: JSON.stringify(payload),
  });

  const text = await response.text();
  let body;
  try {
    body = JSON.parse(text);
  } catch {
    body = { raw: text.slice(0, 240) };
  }

  return {
    ok: response.ok,
    ...event,
    status: response.status,
    errorClass: response.ok ? null : classifyError(response.status, body),
    responseId: body.id || null,
  };
}

Example calls for each tool

import { callVectorEngine } from "./vector-engine-client.js";

const sharedMessages = [
  { role: "system", content: "Answer as a concise engineering assistant." },
  { role: "user", content: "Check whether this route is ready." },
];

for (const tool of ["Dify", "Cursor", "Node.js"]) {
  const result = await callVectorEngine({ tool, messages: sharedMessages });
  console.log(result);
}

How to use the result

If Dify traffic is larger than Cursor traffic, inspect retrieved chunks and prompt templates. If Cursor traffic is larger, inspect code context and hidden instructions. If the Node.js service is larger, inspect history retention and system message composition.

When Vector Engine returns model_not_found, keep the error in the model routing lane. When the local guard returns payload_size_policy, do not spend time changing the Base URL, API Key, or model name. The request did not reach the same failure layer.

This kind of wrapper keeps the OpenAI-compatible API gateway contract small and practical. It also gives the team a shared language: payload issue, key scope issue, Base URL issue, or model route issue.

Registration URL: https://api.vectorengine.cn/register?aff=Igym

在请求到达 Vector Engine 前添加按工具区分的 Payload Size Guard

共享向量引擎时，很多团队会先做一条简单规则：让 Dify、Cursor 和 Node.js 后端都指向同一个 OpenAI 兼容 API 网关，并保持 Base URL 与 model name 一致。这是必要条件，但还不够。长提示词、粘贴的日志、过大的知识库召回内容，以及工具生成的中间输出，都可能在请求到达 LLM API provider layer 前改变请求体。

这篇教程在 Node.js 里加一个小型 payload size guard。它不是为了阻断正常使用，而是记录消息大小、工具来源、模型名和明确的失败类别，避免把本该提前裁剪的请求误判为 model_not_found。

为什么大小检查应该靠近客户端

Dify 可能会追加知识库召回片段。Cursor 可能会带上代码上下文。Node.js 服务可能会附加用户历史或系统指令。即使这些流量都通过向量引擎API中转站，provider route 是共享的，请求形态仍然是每个工具各不相同。

按工具记录大小，可以得到这些信号：

信号	工程用途
工具名称	区分 Dify、Cursor 和服务端流量
Base URL	确认客户端指向预期的向量引擎端点
model name	让模型路由保持可见
payload bytes	判断请求是否在 API 调用前膨胀
error class	区分大小策略、`model_not_found` 和鉴权失败

Node.js 包装器

英文部分给出了完整代码。关键逻辑是：先计算请求体字节数，如果超过阈值，就返回 payload_size_policy，而不是继续调用 API中转站。只有请求体在阈值内，才真正向 chat/completions 发起请求。

三类工具的调用示例

可以用同一组 messages 分别模拟 Dify、Cursor 和 Node.js 的调用来源。真实接入时，把工具名替换成自己的应用名，并把日志写入普通应用日志即可，不要写入 API Key。

如何使用结果

如果 Dify 流量明显大于 Cursor，优先检查知识库召回片段和提示词模板。如果 Cursor 流量较大，检查代码上下文和隐藏指令。如果 Node.js 服务较大，检查历史保留策略和系统消息拼接方式。

当向量引擎中转站返回 model_not_found 时，把问题留在模型路由排查路径里。当本地 guard 返回 payload_size_policy 时，不要浪费时间修改 Base URL、API Key 或 model name，因为请求还没有进入同一层失败路径。

这种封装能让 OpenAI 兼容 API 网关的使用约定更清晰，也能让团队用同一套语言讨论问题：payload 问题、Key 权限问题、Base URL 问题，或者模型路由问题。

注册地址：https://api.vectorengine.cn/register?aff=Igym

DEV Community