DEV Community: Yacine Si Tayeb

Contract-Testing TM Forum Open APIs with Pact + Postman: Stop Breaking Your BSS

Yacine Si Tayeb — Tue, 25 Nov 2025 22:07:50 +0000

Who this is for: platform and QA engineers, API owners, integration leads

What you’ll get: copy-paste Pact contracts, Postman smoke tests and monitors, a GitHub Actions workflow, a crisp backward-compat policy, and optional Pact Broker wiring. Everything below is runnable.

Why Open APIs still break (even when you follow the spec)

TM Forum Open APIs such as TMF620 Product Catalog and TMF622 Product Ordering give teams a common language. That still doesn’t prevent outages. The usual culprits:

Version drift. A provider updates its implementation or CTK; consumers lag. “Optional” fields become practically required; responses change shape; integrations crack.
Schema vs. reality. Specs allow many optional fields. Your consumer needs five of them. A provider “tidies” an enum or omits a field and production breaks. Contracts must reflect consumer expectations, not just a published schema.
Faster release tempo. Teams ship more often; coordination thins; small changes leak.

CTK vs. CDC in one line: CTK proves spec conformance; consumer-driven contracts protect what your consumers actually rely on. You need both.

The fix in practice

Consumer-driven contracts (CDC). Each consumer codifies what it needs; the provider verifies those expectations before shipping. Pact turns that into runnable tests and shareable contracts.

“Contract tests check the shape of external service calls, not the exact data.” — paraphrasing Martin Fowler.

What we’ll build

Pact consumer tests for TMF620 and TMF622 that produce pact files.
Pact provider verification against your mock or service.
Postman collection for smoke tests and a Monitor on a schedule.
GitHub Actions that fail PRs on contract regressions and run smokes after deploy.
Optional Pact Broker to distribute and version contracts across teams.
Pact that actually prevents regressions

TMF620 consumer test: list Product Offerings

// consumer/tmf620.catalog.consumer.spec.js
import { PactV3, MatchersV3 } from '@pact-foundation/pact';
import fetch from 'node-fetch';

const { like, term } = MatchersV3;

const provider = new PactV3({
  consumer: 'CatalogUI',
  provider: 'TMF620-Catalog',
});

describe('TMF620 Catalog - list productOfferings', () => {
  it('returns minimal fields the UI depends on', async () => {
    provider
      .given('product offerings exist')
      .uponReceiving('GET /productOffering?lifecycleStatus=Launched')
      .withRequest({
        method: 'GET',
        path: '/tmf-api/productCatalogManagement/v4/productOffering',
        query: { lifecycleStatus: 'Launched' }
      })
      .willRespondWith({
        status: 200,
        headers: { 'content-type': 'application/json' },
        body: like([{
          id: like('123'),
          name: like('5G 50GB Plan'),
          lifecycleStatus: term({ generate: 'Launched', matcher: '^(Launched|Active)$' }),
          price: like(20),
          currency: term({ generate: 'AUD', matcher: '^[A-Z]{3}$' })
        }])
      });

    return provider.executeTest(async (mock) => {
      const r = await fetch(
        `${mock.url}/tmf-api/productCatalogManagement/v4/productOffering?lifecycleStatus=Launched`
      );
      const json = await r.json();
      // assert only what the UI actually uses
      expect(Array.isArray(json)).toBe(true);
      expect(json[0].name).toBeTruthy();
      expect(json[0].lifecycleStatus).toMatch(/^(Launched|Active)$/);
    });
  });
});

Value: you encode only what the UI relies on. Providers are free to change everything else.

TMF622 consumer test: create Product Order

// consumer/tmf622.order.consumer.spec.js
import { PactV3, MatchersV3 } from '@pact-foundation/pact';
import fetch from 'node-fetch';

const { like, term } = MatchersV3;

const provider = new PactV3({
  consumer: 'CheckoutService',
  provider: 'TMF622-Ordering',
});

describe('TMF622 - create productOrder', () => {
  it('creates an order with minimal required fields', async () => {
    const requestBody = {
      orderItem: [{
        action: "add",
        productOffering: { id: "123" },
        quantity: 1
      }]
    };

    provider
      .given('catalog offering 123 is orderable')
      .uponReceiving('POST /productOrder')
      .withRequest({
        method: 'POST',
        path: '/tmf-api/productOrderingManagement/v4/productOrder',
        headers: { 'content-type': 'application/json' },
        body: requestBody
      })
      .willRespondWith({
        status: 201,
        headers: { 'content-type': 'application/json' },
        body: {
          id: like('po-001'),
          state: term({ generate: 'acknowledged', matcher: '^(acknowledged|inProgress)$' }),
          orderItem: like([{
            id: like('1'),
            action: 'add',
            productOffering: { id: like('123') },
            quantity: like(1)
          }])
        }
      });

    return provider.executeTest(async (mock) => {
      const r = await fetch(`${mock.url}/tmf-api/productOrderingManagement/v4/productOrder`, {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify(requestBody)
      });
      expect(r.status).toBe(201);
      const json = await r.json();
      expect(json.state).toMatch(/^(acknowledged|inProgress)$/);
      expect(json.orderItem?.[0]?.productOffering?.id).toBe('123');
    });
  });
});

Value: you cover the write path. Providers can change non-essential fields, but not the structure your checkout relies on.

Provider verification and a tiny mock

// provider/pact.verify.js
import { Verifier } from '@pact-foundation/pact';

const pactUrls = [
  process.env.PACT_TMF620 || './pacts/catalogui-tmf620-catalog.json',
  process.env.PACT_TMF622 || './pacts/checkoutservice-tmf622-ordering.json'
];

const opts = {
  providerBaseUrl: process.env.PROVIDER_URL || 'http://localhost:3000',
  pactUrls,
  publishVerificationResult: false
};

new Verifier(opts)
  .verifyProvider()
  .then(() => console.log('✅ Provider verified against consumer expectations'))
  .catch((e) => { console.error('❌ Verification failed', e); process.exit(1); });

// provider/mock.js
import express from "express";
const app = express();
app.use(express.json());

// TMF620
app.get("/tmf-api/productCatalogManagement/v4/productOffering", (req, res) => {
  if (req.query.lifecycleStatus !== "Launched") {
    return res.status(400).json({ error: "bad query" });
  }
  res.json([{
    id: "123",
    name: "5G 50GB Plan",
    lifecycleStatus: "Launched",
    price: 20,
    currency: "AUD"
  }]);
});

// TMF622
app.post("/tmf-api/productOrderingManagement/v4/productOrder", (req, res) => {
  const body = req.body || {};
  if (!Array.isArray(body.orderItem) || !body.orderItem[0]?.productOffering?.id) {
    return res.status(422).json({ error: "invalid order" });
  }
  res.status(201).json({
    id: "po-001",
    state: "acknowledged",
    orderItem: [{
      id: "1",
      action: "add",
      productOffering: { id: String(body.orderItem[0].productOffering.id) },
      quantity: Number(body.orderItem[0].quantity || 1)
    }]
  });
});

app.listen(3000, () => console.log("Mock provider on :3000"));

Show a failing case so teams learn to debug fast

Break the mock intentionally. For example, remove currency from the TMF620 response or change state to "created" in TMF622. Provider verification will fail with output like:

❌ Verification failed
- Expected body to have property: currency
- Expected "state" to match /^(acknowledged|inProgress)$/ but was "created"

Takeaway: this is exactly the signal you want in CI before users see a break.

Postman smoke tests and monitors

Contracts protect client expectations; smoke tests protect availability, auth, and latency.

Postman request examples use {{baseUrl}} so you can switch environments.

GET Launched offerings

GET {{baseUrl}}/tmf-api/productCatalogManagement/v4/productOffering?lifecycleStatus=Launched

Tests

pm.test("200 OK", () => pm.response.to.have.status(200));
pm.test("Launched only", () => {
  const body = pm.response.json();
  pm.expect(body.every(x => x.lifecycleStatus === "Launched")).to.be.true;
});
pm.test("p95 under 500ms", () => {
  pm.expect(pm.response.responseTime).to.be.below(500);
});

POST product order

POST {{baseUrl}}/tmf-api/productOrderingManagement/v4/productOrder
Content-Type: application/json

{
  "orderItem": [{
    "action": "add",
    "productOffering": { "id": "123" },
    "quantity": 1
  }]
}

Tests

pm.test("201 Created", () => pm.response.to.have.status(201));
pm.test("State acknowledged/inProgress", () => {
  const j = pm.response.json();
  pm.expect(j.state).to.match(/^(acknowledged|inProgress)$/);
});

Monitor operations that won’t page you at 2am

Create two environments: staging and prod with different baseUrl values.
Route alerts: staging → Slack channel with severity “warn”; prod → PagerDuty or email with severity “page”.
Start with the three most customer-visible flows only. Add more later.

Golden payloads and a backward-compat policy

Pin a golden response per endpoint to a tag.

Example:

// /docs/golden/GET-productOffering-Launched.json
[
  {
    "id": "123",
    "name": "5G 50GB Plan",
    "lifecycleStatus": "Launched",
    "price": 20,
    "currency": "AUD"
  }
]

Backward-compat policy

Additive changes only to existing endpoints.
Any breaking change uses a new versioned path /v2/... or clearly prefixed fields xV2_*.
Dual-write and dual-read for 120 days, then deprecate with docs and monitor alerts.

Compatibility is a process, not a promise.

GitHub Actions you can copy

# .github/workflows/api-quality.yml
name: API Quality

on:
  pull_request:
  push:
    branches: [ main ]

env:
  BASE_URL_STAGING: https://staging.example.com
  BASE_URL_PROD: https://api.example.com

jobs:
  pact-consumer:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
        working-directory: consumer
      - run: npm test -- --runInBand
        working-directory: consumer
      - name: Upload pact artifact
        uses: actions/upload-artifact@v4
        with:
          name: pact
          path: consumer/pacts/*.json

  pact-provider-verify:
    runs-on: ubuntu-latest
    needs: pact-consumer
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - name: Download pact
        uses: actions/download-artifact@v4
        with:
          name: pact
          path: provider/pacts
      - run: npm ci
        working-directory: provider
      - name: Start provider mock
        run: npm run start:mock &
        working-directory: provider
      - name: Verify against pact
        run: node pact.verify.js
        working-directory: provider

  postman-smoke-staging:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Postman collection with Newman (staging)
        uses: matt-ball/newman-action@v1
        with:
          collection: postman/collections/tmf-smoke.json
          environment: postman/envs/staging.json
          reporters: cli,junit

  postman-smoke-prod:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Postman collection with Newman (prod)
        uses: matt-ball/newman-action@v1
        with:
          collection: postman/collections/tmf-smoke.json
          environment: postman/envs/prod.json
          reporters: cli,junit

Security hygiene

Store secrets such as Postman API keys or Pact Broker tokens in GitHub Encrypted Secrets.
Use least-privilege tokens and avoid printing secrets in logs.
Do not run prod monitors on pull requests.

Optional: share contracts via a Pact Broker

If you have multiple teams and pipelines, a Pact Broker makes contract distribution and versioning easier.

Publish consumer pacts on PR merge; tag by env or version.
Providers pull and verify the latest contracts for their tagged stream.
Gate deploys on verification status.

Minimal publishing step:

pact-broker publish consumer/pacts \
  --broker-base-url=$PACT_BROKER_URL \
  --broker-token=$PACT_BROKER_TOKEN \
  --consumer-app-version=$GITHUB_SHA \
  --tag staging

Copy-paste quickstart structure

/contracts
  /consumer   # Pact consumer tests (TMF620, TMF622)
  /provider   # Pact verification + mock
/postman
  /collections
  /envs       # staging.json, prod.json with baseUrl
/docs/golden/GET-productOffering-Launched.json
.github/workflows/api-quality.yml

Start with TMF620 “list productOfferings” and TMF622 “create productOrder”. Add endpoints as you touch them.

SLIs you can measure in a week

Contract verification pass rate per service (target ≥ 99 percent).
Time to detect a breaking change via CI (target under 10 minutes).
Smoke MTTR (time from monitor failure to green).
Change failure rate on API deploys (drop by 30–50 percent after rollout).
Contract age without failure (days since last failing contract).

These are the signals that outages are going down.

Common pitfalls and how to avoid them

Flaky fields such as timestamps and IDs. Use Pact matchers like like and term.
Over-asserting consumers. Assert only what you use; keep contracts small and specific.
“Optional” fields that became required. Capture them in golden payloads and add a consumer contract.
Noisy monitors. Limit to your top three customer-visible flows; align thresholds with SLOs.
Contract sprawl. Keep one contract per consumer use-case per endpoint to avoid brittle mega-contracts.

Where this fits your stack (and how we use it)

If you run a TMF-aligned BSS, contract tests sit between product catalog, ordering, and channels. That’s how we harden Veris Cloud BSS integrations at Cloudnet.ai: schema and contracts on the producer side, smokes and monitors on the consumer side.

For teams operating both BSS and private 5G, CloudRAN.AI follows the same API quality bar.

Closing thought

Contract tests aren’t paperwork. They’re how independent teams ship faster without collateral damage. Wire Pact into TMF620 and TMF622, keep Postman monitors watching real endpoints, and enforce a simple compatibility policy. You’ll see fewer Friday-night incidents, fewer “what changed?” threads, and calmer releases.

Measuring End-to-End Latency for Robots and Cameras Over 5G (Without RF Gear)

Yacine Si Tayeb — Wed, 12 Nov 2025 02:12:09 +0000

Who it’s for: edge devs, robotics/video teams, SREs

What you’ll get: a repeatable lab you can run on a laptop (no radios), the exact metrics to collect, acceptance targets you can defend, and a simple “latency budget” you can use to argue trade-offs with stakeholders.

Why measure this way?

Robots and camera uplinks don’t fail because the spec was wrong—they fail because your encode/decode, queues, and network jitter don’t line up under real workload.

You can surface those issues without touching spectrum: emulate the 5G user plane with containers, drive realistic traffic, and measure application-level latency. When you later move onto a real 4G/5G testbed, you’ll bring the same tests and thresholds. Same harness in lab and field = honest comparisons.

What “good” looks like

Tele-op/AGV control loops

One-way network path for control: ≤ 20–50 ms (pick for your safety envelope)
Jitter p95: ≤ 5–10 ms
Loss: ≈ 0%
If you only have RTT, budget ≤ 40–100 ms RTT with tight jitter.

Live camera uplink (1080p30 “low-latency”)

Glass-to-glass: ≤ 150–250 ms
Network one-way: ≤ 30–60 ms
Encode / Decode / Render each: ~30–60 ms

These are sane starting points. Tighten once your robot/camera vendor signs off.

The RF-free lab (what to spin up)

You’re not proving RF performance; you’re validating the end-to-end behavior of your workloads over a 5G-like core + user plane.

Core: Open5GS (AMF/SMF/UPF with WebUI + Mongo)
RAN/UE sim: UERANSIM (gNB + UE)
Workloads:
- a tiny UDP echo service for control-loop RTT
- GStreamer sender/receiver for low-latency H.264
- a lightweight timestamp beacon alongside video to approximate one-way network latency
Observability: Prometheus + Grafana (p50/p95/p99 views)

Tip: run the first pass with host networking to avoid NAT surprises. Harden later.

Bring-up (illustrative):

# Start core + gNB/UE simulators (compose bundle you trust)
docker compose up -d open5gs mongo ueransim-gnb ueransim-ue

What to actually measure (and why)

A. App RTT for control

Send a 50–200-byte UDP packet every 10 ms, echo it back, and record RTT histograms. This reflects your control loop far better than ICMP ping.

B. One-way proxy for video path

Emit 30 JSON “beacons”/sec that include a timestamp; receive them next to your video sink. With NTP-synced clocks, now − beacon_ts gives you an upper-bound estimate of one-way network+queueing latency. You don’t need nanosecond precision—directional insight is the win.

C. Throughput/jitter sanity

Fire a short UDP iperf3 run so you know the path isn’t constrained by an obvious bottleneck while you tune buffers/bitrates.

Minimal commands (enough to get signal)

Baseline throughput/jitter

iperf3 -u -b 20M -c 127.0.0.1 -p 5201 -t 30

Low-latency H.264 test stream
Sender to Receiver on localhost (tune bitrate/jitterbuffer to your case):

# Sender (H.264, tuned for low latency)
gst-launch-1.0 videotestsrc is-live=true ! video/x-raw,framerate=30/1 ! \
  x264enc tune=zerolatency speed-preset=ultrafast key-int-max=30 bitrate=2500 ! \
  rtph264pay pt=96 ! udpsink host=127.0.0.1 port=5600

# Receiver (tight jitter buffer)
gst-launch-1.0 udpsrc port=5600 caps="application/x-rtp,media=video,encoding-name=H264,payload=96" ! \
  rtpjitterbuffer latency=50 drop-on-late=true ! rtph264depay ! avdec_h264 ! fakesink sync=true

Control-loop RTT (concept)
Use a tiny UDP echo server/client (netcat or a 20-line script). Log p50/p95/p99 over 5–10 minutes under the same conditions as your video test.`

The Latency Budget (how to argue trade-offs)

Break your end-to-end into chunks:

L_total = L_cam + L_enc + L_net_up + L_core + L_net_down + L_dec + L_render

You will directly measure:

L_app_RTT ≈ 2 × (L_net_up + L_core + L_net_down) via UDP echo
L_oneway_proxy ≈ L_net_up + L_core via timestamp beacons

Example goal: 1080p30 at ≤ 200 ms glass-to-glass

Encode 40 ms, Decode 40 ms, Render 30 ms to ~110 ms non-network
Leaves ~90 ms for the network round-trip
Target p95 app RTT ≤ 90 ms and p95 one-way ≤ 45 ms

If you miss the budget:

Tighten rtpjitterbuffer latency from 50 to 30 ms (watch for drops)
Fix MTU fragmentation (keep payloads < path MTU)
Pin CPU for sender/receiver; disable aggressive power saving
Use CBR at a realistic bitrate; avoid huge GOPs that add burstiness

This is how you make latency a budget conversation, not a hunch.

Dashboards and alerts (two panels, three rules)

Dashboards

Control RTT: p50/p95/p99 over time, plus loss
One-way proxy: p95 over time (directional check for congestion)

Alert thresholds (start conservative)

Control RTT p95 > 80 ms for 5 min - warn; > 100 ms - critical
One-way p95 > 45 ms sustained - warn; > 60 ms - critical
Packet loss > 0.5% - warn; > 1% - critical

These catch regressions without paging you for harmless blips.

Reproducibility checklist (the stuff that ruins results)

Clock sync: host NTP green; containers inherit host time
CPU hygiene: pin cores; keep frequency stable; avoid thermal throttling
Warm-up: discard the first 10–20 seconds (encoders & jitter buffers)
Change control: one knob at a time (bitrate or jitter buffer or GOP)
Background load: note it; don’t compare quiet vs. congested runs
Run twice: your “success” isn’t real if you can’t repeat it within ±10%

What “success” looks like before you touch RF

Robots/AGVs: sustained p95 control RTT ≤ 80–90 ms, p99 ≤ 120 ms, near-zero loss
Cameras (1080p30): one-way p95 ≤ 45–60 ms with jitter buffer ≤ 50 ms, stream clean (no bursty drops)
Repeatability: same numbers (±10%) across three runs at different times

If you can’t hit those in emulation, don’t expect miracles in the field. Fix lab hygiene first.

From laptop to field (same harness, new radio)

When you’re happy with your numbers, move the same workloads and thresholds onto a portable 4G/5G kit. Keep the GStreamer settings, the echo cadence, the dashboards—everything. That’s how you tell if RF and mobility are the newvariables, not your pipeline.

Where our team fits: at CloudRAN.AI we use portable, software-defined 4G/5G kits (COTS servers, cloud-managed) so teams can lift this exact harness into venues, factories, or campuses. Same RTT checks. Same Grafana. Same acceptance thresholds. If you’re also monetizing outcomes, Cloudnet.ai’s BSS (Veris) sits on the back end to launch and meter those workloads once they graduate from lab.

A 90-minute runbook

Start Open5GS + UERANSIM (compose bundle you trust).
Add one test UE in the core UI (IMSI/K/OPC).
Sanity-check with a 30-second iperf3 UDP run.
Start UDP echo + the H.264 sender/receiver.
Turn on your dashboards (p50/p95/p99; one-way p95).
Tune jitter buffer and bitrate until you hit the budget.
Save the dashboard; export “golden” JSON and alert thresholds.
Repeat the run. If repeatability is good, you’re ready for field tests.

Bottom line

You don’t need radios to know if robots and cameras will behave.
Measure control RTT and a one-way proxy under realistic load.
Use a latency budget to make trade-offs explicit.
Add two panels and three alerts to catch regressions quickly.
When you’re green, move the same harness to a portable testbed and validate in the wild.

Delving Deeper: Enriching Microservices with Golang with CloudWeGo

Yacine Si Tayeb — Thu, 22 Feb 2024 07:28:14 +0000

What if there existed an RPC framework that provided not only high performance and extensibility but also a robust suite of features and a thriving community support?

CloudWeGo, a high-performance extensible Golang and Rust RPC framework originally developed and open-sourced by ByteDance, has caught my eye as it fits the bill perfectly.

CloudWeGo VS Other RPC Frameworks

While gRPC and Apache Thrift have served the microservice architecture well, CloudWeGo's advanced features and performance metrics set it apart as a promising open source solution for the future.

Built for the modern development landscape by embracing both Golang and Rust, CloudWeGo delivers advanced features and excellent performance metrics. As proof of its performance, benchmark tests have shown that Kitex surpasses gRPC by over 4 times in terms of QPS and latency, with a throughput increased by 51% - 70% in terms of QPS (Queries Per Second) and latency.

This equips developers with a tool that doesn't just meet but decidedly surpasses the performance requirements of modern microservices. Let's delve into some specific use cases to understand CloudWeGo's potential.

Bookinfo: A Tale of Traffic Handling

Consider the case of Bookinfo, a sample application provided by Istio, rewritten using CloudWeGo's Kitex for superior performance and extensibility.

This use case is illustrative of how traffic-heavy services can significantly benefit from CloudWeGo's performance promise. This integration also demonstrates how CloudWeGo stands above traditional Istio service mesh when it comes to traffic handling and performance.

With Kitex and Hertz handling traffic redirection, the Bookinfo project can manage high traffic volumes efficiently, ensuring swift responses and a better user experience.

import (
  "github.com/cloudwego/kitex/server"
)

func main() {
  svr := echo.NewServer(new(EchoImpl), server.WithName("echo"))
  listener, _ := net.Listen("tcp", ":8888")
  svr.Serve(listener)
}

The above code snippet is a simplified example of how the Bookinfo project can be rewritten using Kitex for better performance.

Easy Note: The Magic of Simplicity

CloudWeGo's commitment to simplifying complex tasks shines in its application to the Easy Note project. It leverages CloudWeGo to implement a full-process traffic lane. The note-taking platform needs to be responsive and efficient, a need fulfilled by CloudWeGo's high-performance networking library, Netpoll.

The integration of CloudWeGo has elevated the Easy Note application to compete effectively with other note-taking platforms, proving how simplicity can indeed lead to power.

import (
  "github.com/cloudwego/kitex/server"
)

type RPCService struct{}

func (s *RPCService) Handle(ctx context.Context, req *Request) (*Response, error) {
  resp := &Response{Message: "Echo " + req.Message}
  return resp, nil
}

func main() {
  rpcHandler := &RPCService{}
  svr := server.NewServer(rpcHandler)
  listener, _ := net.Listen("tcp", ":8888")
  svr.Serve(listener)
}

The snippet above gives a glimpse of how CloudWeGo helps to enhance the efficiency of the Easy Note application.

Book Shop: E-Commerce Made Easy

In the bustling e-commerce landscape, Book Shop stands as a testament to CloudWeGo's capacity for seamless integration. Integrating middleware like Elasticsearch and Redis into a Kitex project to build a solid e-commerce system that rivals more complex platforms.

CloudWeGo's ability to effectively integrate with popular technologies like Elasticsearch and Redis ensures that businesses need not compromise on choosing an open-source RPC framework.

import (
  "github.com/cloudwego/kitex/server"
)

type ItemService struct {}

func (s *ItemService) AddItem(ctx context.Context, item *Item) error {
  // Add to Elasticsearch
  // Add to Redis
  // Return error if any
  return nil
}

func main() {
  itemHandler := &ItemService{}
  svr := server.NewServer(itemHandler)
  listener, _ := net.Listen("tcp", ":8888")
  svr.Serve(listener)
}

The above snippet is a basic representation of how the Book Shop e-commerce system operates with CloudWeGo, Elasticsearch, and Redis.

FreeCar: Driving Innovation

The FreeCar project is an excellent illustration of how CloudWeGo can revamp the operations in a time-sharing car rental system, posing a strong alternative to existing ride-hailing applications.

This real-world implementation demonstrates how CloudWeGo's robust features can optimize operations, fostering efficiency and scalability in industries beyond tech.

import (
  "github.com/cloudwego/kitex/server"
)

type CarService struct {}

func (s *CarService) BookRide(ctx context.Context, rideRequest *RideRequest) (*RideConfirmation, error) {
  // Business logic to handle ride booking
  // Return confirmation or error
  return nil, nil
}

func main() {
  rideHandler := &CarService{}
  svr := server.NewServer(rideHandler)
  listener, _ := net.Listen("tcp", ":8888")
  svr.Serve(listener)
}

The above snippet is a simplified representation of how FreeCar utilizes CloudWeGo.

What Draws Me to CloudWeGo?

As I venture further into the landscape of alternative RPC frameworks, and explore the CloudWeGo project, several factors stand out:

Performance: In the world of microservices, performance could mean the difference between success and failure. CloudWeGo shines when it comes to performance, with QPS and latency scores that leave other RPC frameworks trailing.
Extensibility: As a developer, what you'll appreciate most about Kitex is its promise of extensibility, allowing projects to swiftly adapt to growing demands and complexities.
Robustness: The rich feature set of CloudWeGo, including support for multiple message protocols, transport protocols, load balancing, circuit breakers, and rate limiting, offers an all-inclusive solution for designing and managing microservices.
Community Support: The fact that CloudWeGo is backed by ByteDance assures me of strong community support. The wealth of resources and discussions available can solve common issues and support continuous learning.
Real-world Applications: Practical applications in diverse projects demonstrate CloudWeGo’s versatility and scalability, affirming my trust in its effectiveness.

Embracing the Future of Microservices

With each use case, CloudWeGo's potential becomes increasingly clear. Developers can now build high-performing, extensible, and robust applications, harnessing the true essence of microservices - no matter if they prefer working with Golang or Rust.

If you're considering a new tool for your microservice architecture, especially if you are interested in Rust, give CloudWeGo a try. The future of microservices awaits you.

Enhancing Performance in Microservice Architecture with Kitex

Yacine Si Tayeb — Thu, 01 Feb 2024 16:00:00 +0000

This post explores the creation and optimization of ByteDance’s Remote Procedure Call (RPC) framework, Kitex, to address challenges within microservice architecture. Highlighting innovative performance optimization techniques, controlled Garbage Collection and concurrency adjustment, this insightful read offers hands-on strategies and future plans for evolving high-performance microservice systems.

Introduction

The team at ByteDance initiated the creation of the Remote Procedure Call (RPC) framework, Kitex, alongside several related fundamental libraries in 2019. This endeavor originated from confronting functionality and performance challenges within our extensive microservice architecture. We also wanted to combine the knowledge and insights gathered from previous frameworks. This development project was officially released for open-source contribution on GitHub in 2021.

From 2019 to 2023, our internal microservices have seen substantial growth. During this period, the Kitex framework has undergone numerous cycles of optimization and testing to enhance its performance and efficiency. In this article, we share performance optimization techniques that we've systematically implemented over the past few years.

The Evolution and Status Quo of Kitex

Understanding the Need for an RPC Framework

Although the Remote Procedure Call (RPC) framework has a long history, its wide-scale use as a crucial component aligns with the advent of microservice architecture. Therefore, it's vital to revisit its historical developments and comprehend why an RPC framework is necessary.

Background: Monolithic Architecture Era

In this era, system services exhibited the following features:

Distinct business logic was categorized via functions.
The performance pressure was on the database, prompting the evolution from manually distributed databases to an automated distributed structure.

A typical business coding model during this period looked something like this:

func BuySomething(userId int, itemId int) {
    user := GetUser(userId)
    sth := GetItem(itemId)
}

func GetUser(userId) {
    return db.users.GetUser(userId)
}

func GetItem(itemId) {
    return db.items.GetItem(itemId)
}

This style of coding is straightforward, especially once built on top of a well-structured design pattern, can make it straightforward to refactor and write unit tests. Many IT systems still operate using this architecture.

However, as online businesses rapidly developed, we encountered the following limitations in some of the larger internet projects:

There's a limit to computational power: the maximum computing power of a single request is less than or equal to the total computational power of a single server divided by the number of requests processed simultaneously.
There's a constraint around developmental efficiency: the growth of the code repository, team size, and code complexity do not have a linear relationship. This makes maintenance incrementally more challenging as the business grows, resulting in a more complicated online implementations.

The Shift: Microservice Architecture Era

To overcome the issues inherent in the monolithic architecture, the IT community embarked on a journey into the era of microservice architecture. Here's an example of typical code used in a microservice architecture:

func BuySomething(userId int, itemId int) {
    user := client.GetUser(userId) // RPC call
    sth := client.GetItem(itemId) // RPC call
}

RPCs (Remote Procedure Call) allows business systems to call remote services as though invoking local methods. This reduces the complexity of understanding business operations to its most fundamental form and minimizes changes in business coding habits during the transition from a monolithic architecture to a microservice architecture.

The Cost and Path of Optimizing RPC Performance

Before the introduction of RPC, the sole overhead in the following code is merely a function call, an operation at the nanosecond level, not accounting for inline optimization.

func client() (response) {
    response = server(request) // function call
}

func server(request) (response) {
    response.Message = request.Message
}

Upon transitioning to an RPC call, the overhead directly elevates to the millisecond level, a latency difference of 10^6, which highlights the high cost of RPC and indicates considerable room for optimization.

func client() (response) {
    response = client.RPCCall(request) // rpc call - network
}

func server(request) (response) {
    response.Message = request.Message
}

The complete process of an RPC call is outlined below, and we will elaborate on our performance optimization practices for each step in the sections to follow.

Why Develop Our Custom RPC Framework?

Before delving into performance practices, let's talk about why we opted to develop a new RPC framework. There are numerous existing frameworks available, so why did we need a new one? Some of the primary reasons include:

Internally at our company, we primarily use the Thrift protocol for communication. Most mainstream Go frameworks do not support the Thrift protocol, and extending to support multiple protocols isn't a straightforward task.
Recognizing our company's extremely high performance requirements, we realized that deep optimization across the entire operation chain was essential. The vast scale and complexity of our microservices demanded a bespoke, highly customizable, and scalable framework that could provide such flexibility and meet our rigorous performance standards.

How Does Kitex Compare to Other Frameworks?

Kitex supports both the Thrift and gRPC protocols. Considering the lack of Thrift-compatible frameworks in the Go ecosystem, we used the gRPC protocol for our comparative study with the grpc-go framework. Check out the results:

gRPC Unary Comparison:

gRPC Streaming Comparison:

Performance Optimization Practices within the Kitex Framework

Many performance optimization strategies used in Kitex aren't exclusive to Go; however, for the sake of convenience, we're using Go in our examples. In the following sections, we'll introduce different performance optimization practices applied in Kitex, following the process of a RPC call.

Optimizing Encoding and Decoding

Common Challenges with Encoding and Decoding

Using Protobuf as an example, we encounter the following problems associated with encoding decoding operations:

Computational Overhead:

a. Additional information needs to be retrieved through runtime reflection.

b. There's a need to invoke multiple functions and create several small objects, which adds to the processing overhead.

Garbage Collection (GC) Overhead:

Memory reuse is a significant challenge, which often leads to the overhead of garbage collection operations during the encoding and decoding operations.

Code Generation Optimization: Introducing FastThrift & FastPB

We've introduced encoding and decoding capabilities to Kitex by generating a large volume of code for both Thrift and Protobuf protocols. Since generated code can optimize preset runtime information, we can avoid additional operations during runtime and achieve several benefits.

Memory Reuse and Size Pre-calculation

During serialization, we can invoke Size() at a minimal cost and use it to pre-allocate a fixed-size memory block.

Go code example:

type User struct {
   Id   int32
   Name string
}

func (x *User) Size() (n int) {
   n += x.sizeField1()
   n += x.sizeField2()
   return n
}

// Framework Process
size := msg.Size()
data = Malloc(size) // allocate memory
Encode(user, data) // encoding user object directly into the allocated memory to save one time of copy
Send(data) // send data
Free(data) // reuse the allocated memory at next Malloc

Minimize Function Calls and Object Creation

Reducing the costs of invoking functions and creating small objects can yield significant benefits. This approach is especially beneficial in a language like Go, which heavily utilizes garbage collection (GC).

As depicted below, the underlying fastWriteField function gets inlined during compile-time. As a result, the serialization FastWrite function essentially conducts sequential writing into a fixed piece of memory. A similar approach applies to FastRead.

Go code example:

func (x *User) FastWrite(buf []byte) (offset int) {
   offset += x.fastWriteField1(buf[offset:])
   offset += x.fastWriteField2(buf[offset:])
   return offset
}
// inline
func (x *User) fastWriteField1(buf []byte) (offset int) {
   offset += fastpb.WriteInt32(buf[offset:], 1, x.Id)
   return offset
}
// inline
func (x *User) fastWriteField2(buf []byte) (offset int) {
   offset += fastpb.WriteString(buf[offset:], 2, x.Name)
   return offset
}

Optimization Results

As a result of these optimizations, we managed to improve the optimization output from the previous contribution of 3.58% to a notable 0.98%.

JIT Substitution for Code Generation: Introducing Frugal (Thrift)

After reaping gains from the hardcoded approach, we encountered the following feedback:

The volume of the generated code increases linearly with the growth of fields.
The generated code depends on the user's version of their respective Kitex command-line tool, which can lead to conflicts during collaborations with multiple contributors.

The feedback encouraged us to consider if the previously generated code could be created automatically at runtime. The answer is 'yes' - but it would require the adoption of Just-In-Time Compilation (JIT) as a method of code optimization.

Advantages of JIT

Registers utilization and deeper inlining: this improves the efficiency of function calls.
Core computational functions use fully optimized assembly code, which leads to improved performance.

Optimization Results of JIT

As a result of JIT optimization, we improved the optimization result from 3.58% to an impressive 0.78%.

Comparing Frugal and Apache Thrift

This section presents a performance comparison of Frugal and Apache Thrift in the context of encoding and decoding.

Optimizing Network Library

Native Go Net Challenges in RPC Scenarios

Native Go Net in RPC situations presents the following challenges:

Each connection corresponds to one coroutine - when there are numerous upstream and downstream instances, the sheer number of Goroutines can significantly influence performance. This is particularly detrimental for businesses with intensive instances.
It is unable to automatically detect the connection's shutdown state.
When a struct undergoes NoCopy serialization, the output typically takes the form of a two-dimensional byte array. However, Go's Write([]byte) interface falls short as it does not support handling non-continuous memory data.
Despite being highly compatible, it's provided by Go Runtime and not conducive or suitable for adding new features.

Go Coding Example

name := "Steve Jobs" // 0xc000000020
req := &Request{Id: int32(1), Name: name}

// Encode to [][]byte
[
 [4 bytes],
 [10 bytes], // no-copy encoding, 0xc000000020
]

// Copy to []byte
buf := [4 bytes + 10 bytes] // new address

// Write([]byte)
net.Conn.Write(buf)

Netpoll Optimization Practices

Here are the main areas we focused on for optimization:

Coroutine Optimization: As much as possible, coroutines are reused and the number of connections aren't tied to the number of coroutines.
Buffer Layer: Netpoll supports zero-copy read and write, and it reuses memory to minimize GC overhead during encoding and decoding.
Customization for RPC Small Packets High Concurrency Scenarios: Includes coroutine scheduling optimization, TCP parameter tuning, and more.
Deep Customization for Internal Environments: Includes modifying the Go Runtime to improve scheduling priority and kernel support for batch system calls, among other things.

Communication Layer Optimization

Intra-Machine Communication Optimization: Issues with Communication Efficiency under Service Mesh

After introducing Service Mesh, the business process primarily communicates with another sidecar process on the same machine, which brings in an additional layer of delay.

Traditional Service Mesh solutions commonly hijack iptables to facilitate traffic forwarding to the sidecar process. This could lead to substantial performance loss at all levels. Kitex has carried out several performance optimization attempts at the communication layer and has finally developed a systematic solution.

Optimization of Intra-machine Communication: UDS Replaces TCP

Performance Comparison between UDS and TCP:

======== IPC Benchmark - TCP ========
      Type    Conns     Size        Avg        P50        P99
    Client       10     4096      127μs       76μs      232μs
  Client-R       10     4096        2μs        1μs        1μs
  Client-W       10     4096        9μs        4μs        7μs
    Server       10     4096       24μs       13μs       18μs
  Server-R       10     4096        1μs        1μs        1μs
  Server-W       10     4096        7μs        4μs        7μs
======== IPC Benchmark - UDS ========
      Type    Conns     Size        Avg        P50        P99
    Client       10     4096      118μs       75μs      205μs
  Client-R       10     4096        3μs        2μs        3μs
  Client-W       10     4096        4μs        1μs        2μs
    Server       10     4096       24μs       11μs       16μs
  Server-R       10     4096        4μs        2μs        3μs
  Server-W       10     4096        3μs        1μs        2μs

Our performance test indicates the following:

UDS outperforms TCP in all measurements.
Nonetheless, the extent of improvement is not significant.

Optimization of Intra-Machine Communication: ShmIPC Replaces UDS

In pursuit of enhancing the efficiency of inter-process communication, we developed a communication mode based on shared memory. Shared memory communication throws in the complexity of managing the synchronization of various communication states across different processes.

To tackle this, we utilized our communication protocol and retained UDS as the event notification channel (IO Queue) and shared memory as the data transmission channel (Buffer).

For a detailed technical understanding of shmipc, you can refer to our previously published article: Introducing Shmipc: A High Performance Inter-process Communication Library.

Performance Test:

From Cross-Machine to Intra-Machine Communication: A Pod Affinity Solution

Having previously optimized intra-machine communication, we found that it's limited to the data plane communication between the service process and the Service Mesh.

The peer service is possibly not hosted on the same machine. So, the question arises, how can we optimize cross-machine communication? One innovative approach we're considering is converting cross-machine issues into intra-machine issues.

Achieving this in large-scale microservice communication calls for the cooperation of multiple architectural components. As such, we introduced the pod affinity solution to resolve this challenge:

Container Scheduling Layer Alteration: The container scheduling system will consider the relationship and instance situations of upstream and downstream services. It uses affinity scheduling to, as much as possible, assign the instances of upstream and downstream services to the same physical machine.
Traffic Scheduling Layer Alteration: The service control plane needs to identify which downstreams are connected to a certain upstream container. Bearing in mind the context of global load balancing, it calculates the dynamic weight of accessing downstream instances for each downstream instance, aiming to enable more traffic to facilitate intra-machine communication.
Framework Transformation: Extend custom support for the unique communication method of pod affinity. Based on the calculation results of the traffic scheduling layer, the request is dispatched to either the same machine instance or Mesh Proxy.

Microservice Online Tuning Practices

Apart from performance optimization at the framework level, the business logic itself is a significant contributor to the performance bottleneck. To combat this, we have accumulated several practical experiences and strategies.

Solving Latency Through Automated GC Optimization

Challenges with Go's Native GC Strategy

Go's Garbage Collection (GC) strategy was not specifically designed for microservice scenarios, and thus does not prioritize optimizing latency-sensitive businesses. However, RPC services often require low P99 latency.

The essential principles of Go's GC strategy are as follows:

GOGC Principle:

The GOGC parameter sets a percentile value, defaulting to 100, to calculate the heap size required for the next GC trigger: NextGC = HeapSize + HeapSize * (GOGC / 100). This implies that under default settings, the heap size doubles after the last GC.

As an example, if a service's active memory usage is 100MB, the GC is triggered every time the heap grows to 200MB, which may be unnecessary if this service has 4GB of memory.

Drawbacks:

In microservice environments, the service's memory utilization rate is generally quite low, yet aggressive GC persists.
For RPC scenarios, a sizable number of objects are inherently highly reusable. Performing frequent GC on these reusable objects degrades the reusability rate.

Primary Requirement:

Decrease the frequency of GC and enhance the reuse rate of resources in microservices, while maintaining safe levels of memory consumption.

gctuner: Automated GC Optimization Strategy

Users can manipulate the desired aggressiveness of GC by setting a threshold, which could for example be set to memory_limit * 0.7. If the memory used is below this value, GCPercent is maximized as much as possible.

If memory utilization doesn't reach the set threshold, the GOGC parameter is set to a larger value, whereas if it exceeds the limit, it is set to a smaller value.
Regardless of the situation, GOGC is capped at a minimum of 50 and a maximum of 500.

Advantages:

GC is delayed when memory utilization is low.
It reverts to the native GC strategy when memory utilization is high.

Cautions:

If other processes share the memory resources, ensuring a reserve of memory resources for these different processes is crucial.
Services in which memory is likely to have excessively extreme peak values might not find this strategy beneficial.

gctuner is currently open-sourced on GitHub.

Concurrent Optimization

What Is the Real CPU Utilization? - The Container Deception

apiVersion: v1
kind: Pod
spec:
  containers:
  - resources:
      limits:
        cpu: "4"

The rapid advancement of container technology has largely affected the development of microservices. Currently, the majority of microservices, including numerous databases across the industry, operate within container environments. For the purpose of this discussion, we'll only touch upon mainstream containers based on cgroup technology.

A standard business development model involves developers acquiring a 4-core CPU container on the container platform. Developers usually assume that their program can utilize up to 4 CPUs simultaneously, and adjust their program configuration based on this understanding.

Upon deployment, if you check upon the container and verify with the 'top' command, all indicators seemingly adhere to the 4-core standard:

Even when using cat /proc/cpuinfo for inspection, you'll witness exactly 4 CPUs.

processor        : 0
// ...
processor        : 1
// ...
processor        : 2
// ...
processor        : 3
// ...

However, these are merely illusions concocted by the container to ease your mental load off programming. The underlying reason for creating such an illusion is to ensure traditional Linux Debug tools function seamlessly within the container environment.

Contrarily, container technology based on cgroups imposes limits only on CPU time, not on the number of CPUs. Suppose you log into the machine to verify the CPU number each thread of the process is using. In that case, you might be taken aback to discover that the sum exceeds the CPU limit set for the container:

When a container requests 4 CPU units, it means it can run for an equivalent of 4 CPU time within a computation cycle (typically 100ms). It doesn't imply that it can only use 4 physical CPUs, nor that at least 4 CPUs can be simultaneously utilized by the program. If the usage time surpasses the CPU limit set for the container, all processes within the container are paused until the end of the computation period. This could lead to the program experiencing lagging issues (throttled).

Is Faster Downstream Parallel Processing Always Better? - Concurrency vs. Timeout

Knowing the upper bound for physical parallel computing in a program is considerably high, we can leverage this insight to increase or decrease the number of working threads (GOMAXPROCS) or adjust the degree of concurrency within the program.

Let's consider a calling scenario where the business sends requests to the same upstream with four concurrent processes. Each request upstream requires 50ms of processing time. Based on this, the downstream sets the timeout time to 100ms.

Though this seems reasonable, if, at that time, the upstream had only two CPUs available to handle requests (which also have to manage other work or perform Garbage Collection activities), the third RPC request would likely time out.

However, simply reducing concurrency isn't always the solution and it doesn't benefit all cases.

If upstream's computational resources are abundantly available, increasing concurrency could efficiently utilize the processing power upstream.

Balancing Resource Utilization – Reserving Computational Capacity for Other Processes

If there are multiple processes within the container, it's essential to reserve resources for these operations. This consideration is particularly crucial in scenarios like the deployment of a Service Mesh data plane, where the same container operates as a sidecar.

If a downstream process utilizes the entirety of the time slice allocated within a computation cycle, it's highly probable that it'll face resource throttling when it's the upstream process's turn, which could subsequently affect the service’s latency.

Optimizing Service Concurrency Degree

Adjust the Number of Work Threads: For instance, the GOMAXPROCS directive in Go allows us to modify the number of working threads.
Alter the Concurrency of Requests in the Code: It's essential for businesses to iteratively test and evaluate the trade-off between the latency gains from increasing concurrency and the stability loss at peak levels to determine an optimal concurrency value.
Use Batch Interfaces: If the business scenario allows, replacing the current interface with a batch interface could be a more effective strategy.

Looking into the Future of Optimization

The Final Frontier: Kernel

Currently the only area we haven't explored for optimization is the Kernel.

In online business environments, we often observe that the communication overhead of RPC accounts for more than 20% of the total overhead for services heavy on I/O operations, even after optimizing RPC to the level of intra-machine communication.

At this point, we've optimized inter-process communication to its extremes. If we are to seek further improvements, we need to break through the existing constraints in Linux inter-process communication fundamentally.

We've already made some preliminary strides in this area. We will continue to share updates on this topic in future articles, so stay tuned.

Reassessing the TCP Protocol

In the context of internal data center communication, the TCP protocol displays some limitations:

Given the superior internal network quality and an incredibly low packet loss rate, many designs within TCP appear redundant.
In situations of large-scale point-to-point communication, TCP long connections may inadvertently degrade into short connections.
While the application layer uses "messages" as a unit, TCP data streams don't offer clear message boundaries, which could complicate synchronization and message handling.

This has led us to question whether we need to develop a proprietary data center protocol, better suited to handle RPC communication.

Continuing to Refine Existing Components

When it comes to existing components, we plan to continue our efforts to enhance their performance and applicability:

Frugal, the Thrift JIT Encoder/Decoder:

Introducing support for the ARM architecture.
Optimizing the backend with Static Single Assignment (SSA).
Accelerating performance with Single Instruction, Multiple Data (SIMD) operations.

Netpoll Network Library:

Refactoring interfaces to ensure seamless integration with existing libraries in the Go ecosystem.
Implementing support for Shared Memory Communications over RDMA (SMC-R).

Pod Affinity:

Expanding from same-machine to same-rack granularity, effectively reducing network latency and improving performance.

In this post, we explored optimizing microservices performance using Kitex, the RPC framework developed by ByteDance. We discussed various techniques, from encoding and decoding enhancements, JIT compilation, network library optimization, and communication layer upgrades, to automated GC optimization, concurrent processing strategies, and microservices online tuning practices.

Kitex has demonstrated its ability to outperform other frameworks in testing comparisons, showcasing its strength in handling complex microservice architectures.

We also briefly looked towards future optimizations, including kernel-level improvements, restructuring the TCP protocols, and further refinement of existing components. With continuous learning and improvements, we are driven to unlock the vast potential in microservice performance optimization, taking us one step closer to the realm of real-time computing.

For any questions or discussions, you're welcome to join our community on GitHub or Slack.

A Rust Framework for Cloud Development: Volo

Yacine Si Tayeb — Thu, 18 Jan 2024 17:00:00 +0000

I. Introduction

Every tool in the CloudWeGo open-source ecosystem has been developed with the aim of simplifying and revolutionizing how developers navigate the cloud environment. An essential part of this ecosystem is Volo, a Rust RPC framework designed to provide a seamless and efficient communication infrastructure.

This guide aims to provide in-depth insights into leveraging Volo in your projects. Built with Rust, Volo brings unique features and advantages into the mix.

II. The Power of Rust-Based Volo in the Real World

As a part of the CloudWeGo family, Volo can make a significant impact in real-world applications. Its high-speed processing capabilities, when combined with the safety and concurrency advantages of Rust, can provide an efficient backbone to high-performance web services and applications.

The beauty of Rust, which Volo encapsulates, is its ability to push beyond the performance boundaries typically associated with languages such as Go. While Go is highly efficient, it does reach a performance ceiling that may not lend itself to deep optimization. However, once a finely optimized Go service is rewritten in Rust, the benefits spring into view.

Here, CPU gains generally rise above 30%, some even reaching over 50%. In some cases, a fourfold increase in CPU gains is observed. Memory gains are even more pronounced, regularly topping 50% and in some cases reaching as high as 90%.

Beyond performance, Rust addresses the unpredictable jitter issues brought about by Go's garbage collection (GC). In doing so, it helps businesses significantly reduce timeout/error rates, decrease P99 latency, and improve the service level agreements (SLA) of their offerings.

Consider the infrastructure of an online marketplace's back-end. Volo can facilitate seamless interactions between the users, the product database, and third-party services, making them more efficient and reliable.

Another use case could be in the gaming industry, where Volo can help manage player data, game state, and real-time multiplayer interactions with low-latency and high reliability.

Rust and Go are not adversaries but rather allies that complement each other, leveraging their respective strengths to compensate for any weaknesses. For applications where ultimate performance, low latency, memory bottlenecks, and stability are of paramount importance, even if it comes at the cost of some iteration speed loss, Rust is the go-to choice.

These applications can fully benefit from Rust's, and by extension Volo's, unrivaled performance optimization and security. However, when performance sensitivity takes a backseat to high I/O operations, and when rapid development and iteration receives priority over stability, Go becomes the preferred choice.

Rust's vast applicability doesn't stop at server-side business and architectural domains. Its exploratory and implementation journey extends to areas such as internal safety, kernel development, AI, frontend, and client-side development. As such, Volo, with its Rust foundation, carries this adaptability and flexibility, ready to conquer diverse domains and real-world challenges.

III. Getting Started With CloudWeGo

CloudWeGo provides a robust set of tools to work with, one of which is Volo. Here's how you can kickstart your journey with Volo within the CloudWeGo ecosystem.

Volo

Prerequisites

If you don’t have the Rust development environment set up, please follow Install Rust to download Rustup and install Rust. Volo supports Linux, macOS, and Windows systems by default.

Install the CLI tool

Volo provides CLI tools of the same name for initializing projects, managing IDLs, and more. To install Volo tool, you need to switch to nightly channel and run the following command: rustup default nightly cargo install volo-cli.

Then run: volo help

You should see something similar to the following:

    volo [OPTIONS] <SUBCOMMAND
OPTIONS:
    -h, --help       Print help information
    -n, --entry-name <ENTRY_NAME    The entry name, defaults to 'default'. [default: default]
    -v, --verbose    Turn on the verbose mode.
    -V, --version    Print version information
SUBCOMMANDS: help    Print this message or the help of the given subcommand(s)
    idl     manage your idl

IV. Creating A Sample Project With CloudWeGo

When starting a new project with Volo, the following steps can guide you through the development of basic components.

Thrift project

1. Write IDL

To create a Thrift project, we need to write a Thrift IDL first. In your working directory, execute the following command:

mkdir volo-example && cd volo-example
mkdir idl && vim idl/volo_example.thrift

Then, enter the following content:

namespace rs volo.example

struct Item {
    1: required i64 id,
    2: required string title,
    3: required string content,

    10: optional map<string, string> extra,
}

struct GetItemRequest {
    1: required i64 id,
}

struct GetItemResponse {
    1: required Item item,
}

service ItemService {
    GetItemResponse GetItem (1: GetItemRequest req),
}
Init the server project
volo init volo-example idl/volo_example.thrift

Note: Here we use the init command, followed by the name of our project, which means we need to generate template code. At the end, you need to specify an IDL used by the server.

At this point, our entire directory structure looks like this:

├── Cargo.toml
├── idl
│   └── volo_example.thrift
├── rust-toolchain.toml
├── src
│   ├── bin
│   │   └── server.rs
│   └── lib.rs
└── volo-gen
    ├── Cargo.toml
    ├── build.rs
    ├── src
    │   └── lib.rs
    └── volo.yml

2. Add logic code

Open src/lib.rs and add the method implementation to the impl block:

pub struct S;

impl volo_gen::volo::example::ItemService for S {
    // This is the part of the code we need to add
    async fn get_item(
        &self,
        _req: volo_gen::volo::example::GetItemRequest,
    ) -> core::result::Result<volo_gen::volo::example::GetItemResponse, volo_thrift::AnyhowError>
    {
        Ok(Default::default())
    }
}

3. Execute

cargo update && cargo build

At this point, you will find volo_gen.rs file under OUT_DIR Directory. Then execute the following command to get our server running:

cargo run --bin server

We now have our server running!

gRPC project

1. Write IDL

To create a gRPC project, we need to write a protobuf IDL first. In your working directory, execute the following command:

mkdir volo-example && cd volo-example
mkdir idl && vim idl/volo_example.proto

Then, enter the following content:

syntax = "proto3";
package volo.example;

message Item {
    int64 id = 1;
    string title = 2;
    string content = 3;

    map<string, string> extra = 10;
}

message GetItemRequest {
    int64 id = 1;
}

message GetItemResponse {
    Item item = 1;
}

service ItemService {
    rpc GetItem(GetItemRequest) returns (GetItemResponse);
}

2. Init the server project

volo init --includes=idl volo-example idl/volo_example.proto

Note: Here we use the init command, followed by the name of our project, which means we need to generate template code. At the end, you need to specify an IDL used by the server.

At this point, our entire directory structure looks like this:

├── Cargo.toml
├── idl
│   └── volo_example.proto
├── rust-toolchain.toml
├── src
│   ├── bin
│   │   └── server.rs
│   └── lib.rs
└── volo-gen
    ├── Cargo.toml
    ├── build.rs
    ├── src
    │   └── lib.rs
    └── volo.yml

3. Add logic code

Open src/lib.rs and add the method implementation to the impl block:

pub struct S;

impl volo_gen::volo::example::ItemService for S {
    // This is the part of the code we need to add
    async fn get_item(
        &self,
        _req: volo_grpc::Request<volo_gen::volo::example::GetItemRequest>,
    ) -> core::result::Result<volo_grpc::Response<volo_gen::volo::example::GetItemResponse>, volo_grpc::Status>
    {
        Ok(volo_grpc::Response::new(Default::default()))
    }
}

4. Execute

cargo update && cargo build

At this point, you will find volo_gen.rs file under OUT_DIR Directory. Then execute the following command to get our server running:

cargo run --bin server

If you followed the above steps, you'll now have your server running!

V. Troubleshooting Tips & FAQ

Like any technology, working with Volo might come up with its own set of challenges. Here are some tips to handle common issues:

- Compilation Errors: If you encounter any compilation errors, it's recommended to double-check your Rust environment and Volo setup. Ensure that you have the latest stable version of Rust and that Volo is correctly installed and updated to the latest version.

- Runtime Issues: If your Volo application runs into issues during runtime, investigate the error messages and logs. Volo errors are designed to be descriptive and should guide you towards the problem source.

- Why is the code generated by volo-cli separately split into the volo-gen crate?

This separation is because Rust's compilation operates on a crate-by-crate basis. Creating the generated code as a separate crate allows for better utilization of the compile cache (idl generally doesn't change frequently).

- How compatible is it with Kitex?

Volo is fully compatible with Kitex, the Golang RPC framework, including functionalities like metadata transmission.

VI. Conclusion

This guide provided a comprehensive look into Volo, a powerful Rust-based component of the CloudWeGo ecosystem. With an understanding of how to set up and use Volo in your projects, you're now equipped to harness the speed and efficiency that Volo brings to your cloud development tasks.

As you continue to explore CloudWeGo, keep integrating its powerful features into your projects, and see the transformative impact it can have on your software development process.

Stay curious, keep learning, and don't hesitate to dive deeper into the boundless potential of CloudWeGo. Happy coding!

Mastering Golang Microservices - A Practical Guide: Embrace High-Performance with Kitex and Hertz

Yacine Si Tayeb — Fri, 12 Jan 2024 16:20:54 +0000

I. Introduction

The world of software development is fast-paced, and having reliable and efficient tools makes a significant difference.
This is where CloudWeGo with two of its major sub-projects - Kitex and Hertz, comes into play. A solution with the potential to transform the way developers navigate the cloud environment, thanks to its robust, open-source technology.

Two of its standout components, Kitex and Hertz, are at the center of our focus in this guide. Kitex is an efficient and powerful RPC framework used for communication between microservices, while Hertz aids in the quick and efficient setup of web services and BFF services. Both are designed to simplify and enhance your development efforts.

Our mission in this guide is simple: to facilitate your understanding of CloudWeGo, its powerful features, and how to harness them in your projects with a clear step-by-step handbook.
Whether you are a seasoned developer familiar with open-source technology or a newcomer exploring cloud development, this guide is designed to cater to your needs.

Once done reading, you will be comfortable setting up CloudWeGo, initiating and developing a project, implementing testing, debugging, deploying your applications, and more.
We'll also share some of the best practices when using CloudWeGo to ensure that you are maximizing the potential of the CloudWeGo open-source ecosystem. Let's dive in!

II. Getting Started With CloudWeGo

As key components of CloudWeGo, Kitex & Hertz, are crucial to getting started. Ensuring you have a suitably configured environment with Golang is a pre-requisite. If you are working on a Windows platform, make sure the version of Kitex is v0.5.2 or higher. Hertz, on the other hand, is compatible across Linux, macOS, and Windows systems.

Installing the CLI tool requires confirmation that the GOPATH environment variable is correctly defined and accessible. This is followed by installing Kitex, Thriftgo, and Hertz. The correct setup can be verified by running their respective versions. If you encounter any problems, your troubleshooting should involve a check on the setup of the Golang development environment.

Kitex & Hertz

Prerequisites

Before diving into CloudWeGo development with Kitex & Hertz, make sure you have set up the Golang development environment. Please follow the Install Go guide if you haven't already.

We highly recommend using the latest version of Golang, ensuring compatibility with three most recent minor release versions (currently >= v1.16).

Additionally, make sure that GO111MODULE is set to ON.

Install the CLI tool

Let's start by installing the CLI tools we will be working with.
Ensure the GOPATH environment variable is properly defined (e.g., export GOPATH=~/go), then add $GOPATH/bin to the PATH environment variable (e.g., export PATH=$GOPATH/bin:$PATH). Make sure that GOPATH is accessible.

Next, install Kitex (go install github.com/cloudwego/kitex/tool/cmd/kitex@latest), Thriftgo (for Thrift protocol - go install github.com/cloudwego/thriftgo@latest), and Hertz (go install github.com/cloudwego/hertz/cmd/hz@latest).

Now, if you run kitex --version, thriftgo --version, and hz --version, you should see output indicating the versions of each CLI tool:

$ kitex --version
vx.x.x
$ thriftgo --version
thriftgo x.x.x
$ hz --version
vx.x.x

Note: If you encounter any issues during the installation, it's likely due to gaps in the setup of the Golang development environment. Usually, you can quickly find a solution by searching for the error message online.

III. Creating A Sample Project

Kitex

Get the example

You can simply click here to download the example.
Or you can clone the sample repository git clone https://github.com/cloudwego/kitex-examples.git.

Run the example

Run with go

Change to the hello directory. Hello is a simple example of Kitex using the Thrift protocol. cd kitex-examples/hello
Run server go run .
Run client open another terminal and go run ./client.

Run with Docker

Go to the examples directory cd kitex-examples
Build the example project docker build -t kitex-examples.
Run the server docker run --network host kitex-examples ./hello-server
Run the client Open another terminal and run docker run --network host kitex-examples ./hello-client

Congratulations! You now have successfully used Kitex to complete an RPC.

Hertz

Quick Start

To create a sample project with Hertz, start by creating the hertz_demo folder in the current directory and navigate to that directory. Then, create the main.go file and add the following code:
package main

import (
    "context"

    "github.com/cloudwego/hertz/pkg/app"
    "github.com/cloudwego/hertz/pkg/app/server"
    "github.com/cloudwego/hertz/pkg/protocol/consts"
)

func main() {
    h := server.Default()

    h.GET("/ping", func(c context.Context, ctx *app.RequestContext) {
        ctx.JSON(consts.StatusOK, app.H{"message": "pong"})
    })

    h.Spin()
}

Next, generate the go.mod file (go mod init hertz_demo), then tidy & get dependencies (go mod tidy).

To run the sample code, simply type go run hertz_demo. If the server is launched successfully, you will see the following message:

2022/05/17 21:47:09.626332 engine.go:567: [Debug] HERTZ: Method=GET    absolutePath=/ping   -- handlerName=main.main.func1 (num=2 handlers)
2022/05/17 21:47:09.629874 transport.go:84: [Info] HERTZ: HTTP server listening on address=[::]:8888

You can test the interface by typing curl http://127.0.0.1:8888/ping. If everything is working correctly, you should see the following output:

{"message":"pong"}

Using CLI tool hz

You can also use the Hertz CLI tool to generate a sample project outside of the GOPATH. Procedures include creating an IDL file named hello.thrift, generating the sample code, obtaining the dependencies, and subsequently running the sample code.
Assuming you are working on a folder outside of GOPATH, create an IDL file called hello.thrift:

namespace go hello.world

service HelloService {
    string Hello(1: string name); 
}

Generate or complete the Sample Code using hz new -idl hello.thrift -module hertz_demo.

Note: since you're currently not in GOPATH, you'll need to add -module or -mod flag to specify a custom module name. After execution, a scaffolding of the Hertz project is created in the current directory, with a ping interface for testing.

Get dependencies (go mod tidy), then run the sample code (go build -o hertz_demo && ./hertz_demo).

If the server is launched successfully, you will see the same message as before, and you can test the interface using the same curl command. Congratulations, you've successfully launched the Hertz Server!

IV. Testing and Debugging Your Project

Testing and debugging your project are essential components whether you are working with Kitex or Hertz.
While dealing with Kitex errors, the IsKitexError method in the kerrors package can be used.

The Kitex framework automatically recovers all panics except those occurring within the goroutine created by the business code using the go keyword.

Kitex

Exception Instruction

Check for Kitex errors using kerrors.IsKitexError(kerrors.ErrInternalException). You can check for a specified error type using errors.Is(err, kerrors.ErrNoResolver). Also, note that you can use IsTimeoutError in kerrors to check whether it's a timeout error.

To get detailed error messages, all detailed errors are defined by DetailedError in kerrors. You can use errors.As to fetch specified DetailedError.

For example:

import "errors"
import "github.com/cloudwego/kitex/client"
import "github.com/cloudwego/kitex/pkg/kerrors"
_, err := echo.NewClient("echo", client.WithResolver(nil))
var de *kerrors.DetailedError
ok := errors.As(err, &de)

DetailedError provides the following methods to fetch a detailed message:

ErrorType() error: to get the basic error type
Stack() string: to get the stack (currently only works for ErrPanic)

Handling panic

Panic that occurs in the goroutine created by the business code using the go keyword must be recovered by the business code. To ensure the stability of the service, the Kitex framework will automatically recover all other panics.

While checking for recovered panic in your middlewares, you can use ri.Stats().Panicked():

// After calling next(...) in your middleware:
ri := rpcinfo.GetRPCInfo(ctx)
if stats := ri.Stats(); stats != nil {
    if panicked, err := stats.Panicked(); panicked {
      // err is the object kitex get by calling recover()
    }
}

FAQ & Answers

Q1: Not enough arguments problem when installing the code generation tool

Please try:
go mod：GO111MODULE=on go get github.com/cloudwego/kitex/tool/cmd/kitex@latest

Q2: Why does set in IDL become slice in generated codes?

Due to JSON serialization, the official Apache Thrift changed the generation type of set from map to slice starting from v0.11.0. To ensure compatibility, Kitex follows this rule.

Q3: Why is there an underscore after some field names?

The official implementation of Thrift forbids identifiers ending in "Result" and "Args" to avoid naming conflicts. When the type name, service name, and method name in the Thrift file start with "New" or end with "Result" or "Args", an underscore is automatically added at the end of the name.

Q4: Does the code generated by a new interface overwrite handler.go?

Generated code under kitex_gen/ will be overwritten. However, handler.go of the server will not be overwritten; new methods will be added correspondingly.

Q5: "Not enough arguments in call to iprot.ReadStructBegin when compiling Thrift interface

Kitex is based on Apache Thrift v0.13 and cannot be directly upgraded since there is a breaking change in Apache Thrift v0.14. Such issues usually arise if a new version of Thrift is pulled during upgrades.

We recommend against using -u parameters during upgrades. You can run the following command to fix the version: go mod edit -replace github.com/apache/thrift=github.com/apache/thrift@v0.13.0

Hertz

Error Type & Error Chain

To handle errors more effectively, Hertz has predefined several error types:

ErrorTypeBind: Error in binding process
ErrorTypeRender: Error in rendering process
ErrorTypePrivate: Hertz private errors that business doesn't need to be aware of
ErrorTypePublic: Hertz public errors that require external perception as opposed to Private
ErrorTypeAny: Other Error

Users should define corresponding errors according to these error types.
In addition to error definition conventions, Hertz also provides ErrorChain capability to make it easier for businesses to bind all errors encountered during request processing to an error chain.

The corresponding API for this is RequestContext.Error(err). Calling this API will tie the err to its corresponding request context. To get all the errors bound by the request context, use RequestContext.Errors.

FAQ & Answers

Q1: High Memory Usage

Connections not Closing due to Client Non-standard Usage: If the client initiates a large number of connections without closing them, there can be a significant waste of resources over time, causing high memory usage problems.
To resolve this, configure idleTimeout reasonably. Hertz Server will close the connection to ensure the server's stability after the timeout. The default configuration is three minutes.

Q2: Vast Request/Response

If the request and response are vast, the data will enter memory, causing significant pressure, especially when stream and chunk are not used. To resolve this, for very vast requests cases, use a combination of streaming and go net.

Q3: Common Error Code Checking

The following error codes are commonly reported by the framework:

404 (Access to the wrong port or No routes matched)
417 (The server returns false after executing the custom ContinueHandler)
500 (Throwing the panic in middleware or in handlerFunc)

For more details and solutions on these and other error codes, please refer to the Kitex User Guide.

Context Guide

Hertz also provides a standard context.Content and a request context as input arguments in the function in the HandleFunc Design. The handler/middleware function signature is:

type HandlerFunc func(c context.Context, ctx *app.RequestContext)

Metadata Storage

Both contexts (c and ctx) have the ability to store values. The choice of which one to use depends on the life cycle of the stored value and the selected context should match. The ctx is primarily used to store request-level variables, which are recycled after the request ends.

It is characterized by high query efficiency (the bottom is map), unsafe coroutines and doesn't implement the context.Context Interface. The c is passed as the context between middleware/handler. It has all the semantics of context.Content, is safe for coroutines, and all that requires the context.Content interface as input arguments can just pass c directly.

V. Observability

Monitoring your application is critical. Both Kitex and Hertz provide a Tracer interface that can be implemented for efficient application monitoring. You can make the most of the numerous instrumentation controls and logging capabilities on offer.

Note: As a framework, it runs with business services. Once the code of services is built, it can be deployed at virtual machines, bare metal machines, or Docker containers as it should be.

Kitex

Configuration and options

For more details, please check server option, client option, and call option.

Observability

Instrumentation Control

Kitex supports flexible enabling of basic and fine-grained Instrumentation. This includes a stats level, client tracing stats level control, server tracing stats level control, and more. For more details, please refer to the Kitex User Guide.

Logging

Kitex supports default logger implementation, injection of custom loggers, and redirection of default logger output. For more details, instructions, and examples, please refer to the Kitex User Guide.

Tracing

Kitex’s OpenTelemetry extension provides support for tracing. For more details, instructions, and examples, please refer to the Kitex User Guide.

Monitoring

The framework doesn’t provide any monitoring, but it provides a Tracer interface. This interface can be implemented by yourself and be injected via WithTracer Option. For more details, instructions, and examples, please refer to the Kitex User Guide.

Hertz

Configuration and options

For more details, please check the configuration instructions.

Observability

Instrumentation

Hertz supports flexible enabling of basic and fine-grained Instrumentation. This includes a stats level, stats level control, and more. For more details, please refer to the Hertz User Guide.

Log

Hertz provides a default way to print logs in the standard output. It also provides several global functions, such as hlog.Info, hlog.Errorf, hlog.CtxTracef, and more, which are implemented in pkg/common/hlog, to call the corresponding methods of the default logger. For more details, instructions, and examples, please refer to the Hertz User Guide.

Tracing

In microservices, link tracing is a very important capability, which plays an important role in quickly locating problems, analyzing business bottlenecks, and restoring the link status of a request.
Hertz provides the capability of link tracking and also supports user-defined link tracking. For more details, instructions, and examples, please refer to the Hertz User Guide.

Monitoring

VI. Best Practices for Developing with CloudWeGo

For a real-world application of Kitex and Hertz, you can explore projects like Bookinfo, Easy Note, and Book Shop. Each of these scenarios demonstrate different business scenarios and use-cases for various CloudWeGo subprojects.

Whether you're dealing with merchant or consumer management, notes maintenance, or integrating different middleware, these projects provide valuable insights into the powerful capabilities of Kitex and Hertz in different contexts.

This guide provides a comprehensive exploration of CloudWeGo's powerful capabilities, particularly its subprojects, Kitex and Hertz. You now have a solid understanding of how to harness these tools effectively in your development projects.

As you continue delving into CloudWeGo, remember to mix the tool's powerful features with your creativity for impressive results in your software development journey.

Stay curious, keep exploring, and stay tuned for our upcoming Rust-focused Volo guide, which will introduce you to yet another exciting aspect of CloudWeGo. Happy coding!

What is Middleware Anyway?

Yacine Si Tayeb — Thu, 30 Dec 2021 09:11:10 +0000

What is and what is Middleware for?

Middleware software enables communication or connectivity between two or more applications or application components in a distributed network. By making it easier to connect applications that weren't designed to connect with one another - and providing functionality to connect them in intelligent ways - Middleware "glues together" separate, often complex and already existing programs to streamline application development and speeds time to market.

There are many types of middleware. Some, such as message brokers or transaction processing monitors (e.g. Amazon Web Services Simple Queue Service, MicrosoftTransactionServer or TuxedoMonitor for the Unix environment), focus on one type of communication. Others, such as web application servers or mobile device middleware (e.g. vCenter Server or Apache Tomcat), provide the full range of communication and connectivity capabilities needed to build a particular type of application. Still others such as SphereEx’s cloud-based integration platform as a service (iPaaS) offering or an enterprise service bus (e.g. IBM App Connect or Microsoft Azure), function as a centralized integration hub for connecting all the components in an enterprise.

Key Advantages:

The advantages brought by the implementation of Middleware solutions could be resumed into the three ways an integrated software layer can deliver a competitive edge. Nowadays customers interact with organizations through an increasing number of technology interfaces-including mobile, cloud, and onsite applications.

These applications are generally built individually and reactively, making it difficult for businesses to react rapidly to business changes caused by everchanging customer demands, market fluctuations or regulatory changes. Middleware technologies can help by providing a flexible layer between applications and technologies.

This layer of flexibility is represented by software that creates a common platform for all interactions internal and external to the organization- system-to-system, system-to-database, human-to-system, Web-based, and mobile-device-based interactions.

Successful organizations recognize the advantages of this service-oriented architecture and implement it intelligently in their enterprises, generally reaping three major benefits.

Enhanced Efficiency

Successful organizations continuously streamline the way they do business by automating their business processes with technology. Middleware technology allows for process automation – such as product ordering and configuration - and reduces costs that would come with a staff member performing the setup manually. This improved cycle time ultimately increases the total volume of business because of the simplified customer interactions.

Boosted Agility

Delivering services across the mobile, cloud, and traditional application platforms is a major challenge. As services converge, customers expect a common and improved user experience. The current IT landscape needs to be agile to meet these requirements. Middleware technology can help reconcile legacy IT systems into reusable, general-purpose functionality blocks that facilitate quicker changes to business processes. As a result, the business is better supported for changes in products and services as well as the introduction of new channels.

ShardingSphere users including China Minsheng Banking Corp., JD.com, Trip.com, iQiyi, OPPO, Vivo and TCL provide feedback showing faster time to market for products and services where Middleware can aid the process.

Fast Innovation

Product development cycles need to be shorter to introduce new, innovative products before competitors. An IT landscape of reusable software services helps the business roll out new products and services faster while maintaining lower development costs. SphereEx and ShardingSphere users can achieve a reduction in total cost of ownership for their existing environments—freeing up valuable resources to invest in new product development.

To get the most out of middleware technology, business and IT team managers should choose the best middleware solution that is complete, integrated, and pluggable.

Business leaders should view their work with their middleware partner as a strategic partnership that simplifies processes. IT managers can require certified integrations between middleware, database, and applications or choose an open source solution such as ShardingSphere that if fully customizable and allows them to implement the middleware products right into their existing environment and reduce costs. The right middleware technology enables an enterprise to create and run agile, intelligent business applications while maximizing IT efficiency through full utilization of modern hardware and software architectures.

ShardingSphere is a data and SQL control and management layer connecting applications and databases, introducing a new ecosystem concept that goes far beyond the traditional Middleware concept and capabilities.

As the commercial company behind the ShardingSphere open source project development, SphereEx provides enterprise solutions and support that better meet the needs of partners that require enterprise-level support services. Concurrently, SphereEx continues to uphold its commitment to the Apache ShardingSphere project and the open source community – always striving for openness, diversification, co-development, and linking world developers and users.