DEV Community: jinhua luo

Asynchronous nginx response header/body filter

jinhua luo — Fri, 05 Jan 2024 07:40:09 +0000

As we all know, due to the design limit, nginx header/body filters are synchronous and you should not perform blocking operations such as blocking system calls, decrypting, decompressing, or forwarding responses to other servers for further processing.

In contrast, envoy supports asynchronous HTTP filters on the response path.

However, there are some use cases where the response headers and body from the upstream server must be processed asynchronously before being sent downstream, such as by requesting a Vault server to verify the digest of the body, or by scanning the content on a dedicated server for confidentiality reasons.

How to interpret the upstream response asynchronously? And what’s the performance?

Please visit my blog for more details:

http://luajit.io/posts/asynchronous-lua-filter-for-nginx-response/

burl: A simple but flexible HTTP/3 testing framework based on bash and curl

jinhua luo — Fri, 05 Jan 2024 07:37:37 +0000

A few months ago, when I ported the QUIC patches from nginx mainline to APISIX and tried to test it, I found that test::nginx didn’t work very well. It uses the wrong listen directive parameter “http3” instead of “quic” (probably due to version differences).

So I was wondering if I could design a simple testing framework because the latest curl already fully supports HTTP/3.

Many testing frameworks prefer DSLs (Domain Specific Languages) such as test::nginx and Hurl because it is more user-friendly. And it looks more data-driven. However, this is not good for developers looking to extend its functionality as they would need to learn the language behind it and take the time to understand the core and extension APIs by reading the code base of this testing framework. In other words, DSL and the language behind it create a knowledge gap for developers because it is not intuitive and cannot map the functions corresponding to the DSL to the language itself. Especially when the framework itself lacks capabilities and needs to be expanded, this gap will bring significant learning costs.

Take test::nignx as an example, which is implemented in Perl. When I need to verify the response body and decrypt it by pre-shared key, The built-in function doesn’t help as it only supports regular expression matching. So I need to write a bunch of Perl code to do this. However, I’m not familiar with Perl, so this isn’t an easy task. As we all know, Perl is not a mainstream language and has a steep learning curve. At that time, I was thinking, how great it would be if I could write code in Python.

The expressive capabilities of test::nginx are limited, even though it already supports sending requests using curl, because it is not a scripting system (Of course, you can extend it using Perl modules, but due to the limitations of the framework design, this is a difficult task for most people).

It assumes your input is a text string, you cannot use other formats such as protobuf, and you cannot compress or encrypt text.
The only output post-processing supported is regular matching, so you cannot parse and verify other forms of content, such as decryption.
You cannot describe complex test procedures, such as sending three requests to trigger the request quota and verify the rate-limiting results.
You cannot make other types of requests such as GRPC, or SOAP.
In my practice, when I have such needs that exceed test::nginx’s capabilities, I have to delegate them to the lua part, which is very troublesome.

For example, when I need to parse and verify the correctness of the CAS Auth plugin, it involves a lot of client-side CAS protocol details, so I have to implement the entire logic in the lua part, which increases the difficulty of maintaining this mix. As we all know, neither Perl nor Lua are popular languages. ;-(

You can see the test case below involves additional lua lib development to test CAS. And, to accommodate test::nginx’s lack of response parsing, you have to do the conversion in the Lua environment. Quite annoying, at least to me, because I’ve written a lot of test cases like this over the years.

=== TEST 2: login and logout ok
--- config
    location /t {
        content_by_lua_block {
            local http = require "resty.http"
            local httpc = http.new()
            -- Test-specific CAS lua lib
            local kc = require "lib.keycloak_cas"

            local path = "/uri"
            local uri = "http://127.0.0.1:" .. ngx.var.server_port
            local username = "test"
            local password = "test"

            local res, err, cas_cookie, keycloak_cookie = kc.login_keycloak(uri .. path, username, password)
            if err or res.headers['Location'] ~= path then
                ngx.log(ngx.ERR, err)
                ngx.exit(500)
            end
            res, err = httpc:request_uri(uri .. res.headers['Location'], {
                method = "GET",
                headers = {
                    ["Cookie"] = cas_cookie
                }
            })
            assert(res.status == 200)
            ngx.say(res.body)

            res, err = kc.logout_keycloak(uri .. "/logout", cas_cookie, keycloak_cookie)
            assert(res.status == 200)
        }
    }
--- response_body_like
uri: /uri
cookie: .*
host: 127.0.0.1:1984
user-agent: .*
x-real-ip: 127.0.0.1

Similarly, Hurl also uses a DSL to describe tests. Its backend language is Rust. So in terms of scalability, it seems to be worse than test::nginx because it needs to compile the code. However, Rust’s ecosystem is much better than Perl’s. So you can write code easily. But anyway, you need to write the code in a different language than the DSL.

Is there a simple testing framework that can satisfy the following requirements?

No DSL, just simple shell scripts
Easy to extend and programming language agnostic
Can be used to initialize and test any server, not limited to nginx, such as envoy

Please visit my blog for more details:
http://luajit.io/posts/a-simple-http3-testing-framework-based-on-bash-and-curl/

Implement grpc client in rust language for openresty/Nginx

jinhua luo — Sun, 27 Nov 2022 07:20:51 +0000

Back in 2019, when I work as individual postgresql/nginx consultant.
Many clients ask me if they could do hybrid programming in nginx.

They said, nginx/openresty is powerful and high performance, but the most significant
drawback is it lacks of ecosystem of mainstream programming languages, especially
when they have to interact with popular frameworks, e.g. kafka, consul, k8s, etc.
The C/Lua ecosystem is slim, even if they bite the bullet and write down C/Lua modules,
they face the awkward fact that most frameworks do not provide C/Lua API.

As known, nginx is multi-proceesses and single threading. In the main thread, it does
nonblocking async event processing. Openresty wraps it in Lua API, combines the coroutine
and epoll into coscoket concept.

Meanwhile, each mainstream programming language has its own threading model. So the problem is,
how to couple with them in coroutine way? For example, when I call the function in Rust
(maybe runs in a thread, or maybe in tokio async task, whatever), how to make it nonblocking in
Lua land, and make the call/return in yield/resume way?

Of course, the traditional way is to spawn an individual process to run Rust,
and at the lua side, use unix domain socket to do the communication, as known as proxy model.
But that way is low-efficient (either development or runtime).
You need to encode/decode the messsage in TLV (type-length-value) format,
and, you need to maintain nginx and proxy process separately, as well as failover and load-balance topics.

Could I make it simple like FFI call but with high performance?
The answer is yes, I created lua-resty-ffi.

lua-resty-ffi

https://github.com/kingluo/lua-resty-ffi

lua-resty-ffi provides an efficient and generic API to do hybrid programming in openresty with mainstream languages
(Go, Python, Java, Rust, etc.).

Features:

nonblcking, in coroutine way
simple but extensible interface, supports any C ABI compliant language
once and for all, no need to write C/Lua codes to do coupling anymore
high performance, faster than unix domain socket way
generic loader library for python/java
any serialization message format you like

Architecture

The combination of library and configuration would init a new runtime,
which represents some threads or goroutines to do jobs.

lua-resty-ffi has a high performance IPC mechanism, based on request-response model.

The request from lua is appended to a queue. This queue is protected by pthread mutex.
The language runtime polls this queue, if it's empty, then wait on the queue via pthread_cond_wait().
In busy scenario, almost all enqueue/dequeue happens in userspace, without futex system calls.

lua-resty-ffi makes full use of nginx nonblocking event loop to dispatch the response from the language runtime.
The response would be injected into the global done queue of nginx, and notify the nginx main thread via eventfd
to handle it. In main thread, the response handler would setup the return value and resume the coroutine
waiting on the response.

As known, linux eventfd is high performance. It's just an accumulated counter, so multiple responses would be folded into one event.

Both request and response data are exchanged in userspace.

What it looks like?

Take python as example.

ffi/echo.py

from cffi import FFI
ffi = FFI()
ffi.cdef("""
char *strdup(const char *s);
void* ngx_http_lua_ffi_task_poll(void *p);
char* ngx_http_lua_ffi_get_req(void *tsk, int *len);
void ngx_http_lua_ffi_respond(void *tsk, int rc, char* rsp, int rsp_len);
""")
C = ffi.dlopen(None)

import threading

class State:
    def poll(self, tq):
        while True:
            task = C.ngx_http_lua_ffi_task_poll(ffi.cast("void*", tq))
            if task == ffi.NULL:
                break
            req = C.ngx_http_lua_ffi_get_req(task, ffi.NULL)
            res = C.strdup(req)
            C.ngx_http_lua_ffi_respond(task, 0, res, 0)
        print("exit python echo runtime")

def init(cfg, tq):
    st = State()
    t = threading.Thread(target=st.poll, args=(tq,))
    t.daemon = True
    t.start()
    return 0

Calls it like normal function call in your lua coroutine:

local demo = ngx.load_ffi("ffi_python3", "ffi.echo?,init",
    {is_global = true, unpin = true})
local ok, res = demo:echo("hello")
assert(ok)
assert(res == "hello")

Benchmark

Send 100,000 requests, in length of 512B, 4096B, 10KB, 100KB respectively.
The result is in seconds, lower is better.

You could see that lua-resty-ffi is faster than unix domain socket, and the difference
is proportional to the length.

Check benchmark in detail.

Develop GRPC client in Rust

Let's use lua-resty-ffi to do some useful stuff.

In k8s, GRPC is standard RPC. How to make GRPC calls in lua?

Let's use Rust to develop a simple but complete GRPC client for lua,
covering unary, client/server/bidirectional streaming calls, as well as TLS/MTLS.

In this example, you could also see the high development efficiency using lua-resty-ffi.

Check https://github.com/kingluo/lua-resty-ffi/tree/main/examples/rust for code.

tonic

tonic is a gRPC over HTTP/2 implementation focused on high performance, interoperability,
and flexibility. This library was created to have first class support of async/await
and to act as a core building block for production systems written in Rust.

tonic is my favourite rust library.
It's based on hyper and tokio.
Everything works in async/await way.

Use low-level APIs

Normal steps to develop grpc client:

write a .proto file
compiles it into high-level API for use, via build.rs
calls generated API

For example:

helloworld.proto

package helloworld;

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}

client.rs

let request = tonic::Request::new(HelloRequest {
    name: "Tonic".into(),
});
let response = client.say_hello(request).await?;

But it's not suitable for lua, because it's expected to call arbitrary protobuf interface
and does not need to compile .proto file.

So we have to look at low-level API provided by tonic.

Let's check the generated file:

helloworld.rs

#[derive(Clone, PartialEq, ::prost::Message)]
pub struct HelloRequest {
    #[prost(string, tag="1")]
    pub name: ::prost::alloc::string::String,
}
...
pub async fn say_hello(
    &mut self,
    request: impl tonic::IntoRequest<super::HelloRequest>,
) -> Result<tonic::Response<super::HelloReply>, tonic::Status> {
...
    let codec = tonic::codec::ProstCodec::default();
    let path = http::uri::PathAndQuery::from_static(
        "/helloworld.Greeter/SayHello",
    );
    self.inner.unary(request.into_request(), path, codec).await
}

message encode/decode

You could see that the request structure is marked with ::prost::Message attribute.
And it uses tonic::codec::ProstCodec::default() to do the encoding.

This part could be done in lua using lua-protobuf.
It could do encode/decode based on the .proto file, just like JSON encode/decode.

Check grpc_client.lua for detail.

But wait, if you do encoding in lua, then how to tell rust to bypass it?

The answer is customized codec, instead of ProstCodec, so that we could transfer the byte array "AS IS".

struct EncodedBytes(*mut c_char, usize, bool);
...
impl Encoder for MyCodec {
    type Item = EncodedBytes;
    type Error = Status;

    fn encode(&mut self, item: Self::Item, dst: &mut EncodeBuf<'_>) -> Result<(), Self::Error> {
        let slice = unsafe { slice::from_raw_parts(item.0 as *const u8, item.1) };
        dst.put_slice(slice);
...
    }
}

impl Decoder for MyCodec {
    type Item = EncodedBytes;
    type Error = Status;

    fn decode(&mut self, buf: &mut DecodeBuf<'_>) -> Result<Option<Self::Item>, Self::Error> {
        let chunk = buf.chunk();
        let len = chunk.len();
        if len > 0 {
            let cbuf = unsafe { libc::malloc(len) };
            unsafe { libc::memcpy(cbuf, chunk.as_ptr() as *const c_void, len) };
            buf.advance(len);
            Ok(Some(EncodedBytes(cbuf as *mut c_char, len, false)))
...
    }
}

For encode, we put the original byte array encoded by lua into the request.

For decode, we copy the byte array to lua and let lua decode it later.

Then, we could use unary directly.

connection management

Each GRPC request is called upon the connection object (interestingly, the connection in tonic has retry mechanism).
I use hashmap to store connections, and assign each connection a unique string, so that the lua side
could index it. In Lua, you could explicitly close it or let the lua gc to handle the close.

Each connection could be configured with TLS/MTLS info, e.g. CA certificate, private key.

message format of request and response

Now, let's look at the message format between lua and rust, which is most important part.

request

#[derive(Deserialize, Debug)]
pub struct GrpcCommand {
    cmd: i32,
    key: String,
    host: Option<String>,
    ca: Option<String>,
    cert: Option<String>,
    priv_key: Option<String>,
    path: Option<String>,
    payload: Option<Base64>,
}

Since we need to handle different kinds of functions, e.g. create a connection, GRPC unary call, etc.
So it's obviously we need a structure to describe all information.

cmd: the command indicator, e.g. NEW_CONNECTION
key: depends on cmd, could be connection URL, connection id or streaming id.
host: HTTP2 host header, used for TLS
ca, cert, priv_key: TLS related stuff
path: full-qualified GRPC method string, e.g. /helloworld.Greeter/SayHello
payload: the based64 and protobuf encoded requset byte array by lua

Note that, all optional fields use Option<>.

JSON is high performance in lua, so no need to worry about the overhead.

Check my blog post
for benchmark between JSON and protobuf if you are interested in this part.

response

This part is more straightforward than the request.
Because for each request type, you could expect only one kind of result.

For example, for NEW_CONNECTION, it returns connection id string;
for UNARY, it returns the protobuf encoded response from the network.

So, returning the malloc() byte array is ok, no need to wrap it in JSON.

GRPC streaming

Under the hood, no matter which type the call is,
unary request,
client-side streaming,
server-side streaming or
bidirectional streaming,
tonic encapsulates the request and response into
Stream object.

So we just need to establish the bridge between lua and Stream, then we could support GRPC streaming in lua.

send direction

We create a channel pair, wrap the receiver part into request Stream.

let (send_tx, send_rx) = mpsc::unbounded_channel::<EncodedBytes>();
let send_stream = tokio_stream::wrappers::UnboundedReceiverStream::new(send_rx);
...
let mut stream = cli
    .streaming(tonic::Request::new(send_stream), path, MyCodec)
    .await
    .unwrap()
    .into_inner();

Then the call upon the stream would be transfer to the network directly.

recv direction

We create an async task to handle the response retrieval.

tokio::spawn(async move {
...
    while let Some(task) = recv_rx.recv().await {
        if let Some(res) = stream.message().await.unwrap() {
            unsafe {
                ngx_http_lua_ffi_respond(task.0, 0, res.0, res.1 as i32);
            }
...
    }
});

close the send direction

Thanks to the channel characteristic, when we drop the send part of the channel pair,
the receiver part would be closed and in turn close the send direction of GRPC stream.

Final look

Unary call:

local ok, conn = grpc.connect("[::1]:50051",
    {
        host = "example.com",
        ca = grpc.readfile("/opt/tonic/examples/data/tls/ca.pem"),
        cert = grpc.readfile("/opt/tonic/examples/data/tls/client1.pem"),
        priv_key = grpc.readfile("/opt/tonic/examples/data/tls/client1.key"),
    }
)
assert(ok)

local ok, res = conn:unary(
    "/grpc.examples.echo.Echo/UnaryEcho",
    {message = "hello"}
)
assert(ok)
assert(res.message == "hello")

conn:close()

Streaming calls:

local ok, conn = grpc.connect("[::1]:10000")
assert(ok)

-- A Bidirectional streaming RPC.
local ok, stream = conn:new_stream("/routeguide.RouteGuide/RouteChat")
assert(ok)

for i=1,3 do
    local note = {
        location = {latitude = 409146138 + i*100, longitude = -746188906 + i*50},
        message = string.format("note-%d", i),
    }
    local ok = stream:send(note)
    assert(ok)

    local ok, res = stream:recv()
    assert(ok)
    ngx.say(cjson.encode(res))
end

local ok = stream:close()
assert(ok)

local ok = conn:close()
assert(ok)

Benchmark

The most important question is, since we wrap the tonic, so how much is the overhead?

I use AWS EC2 t2.medium (Ubuntu 22.04) to do the benchmark.

Send 100,000 calls, use lua-resty-ffi
and tonic helloworld client example respectively.

The result is in seconds, lower is better.

You could see that the overhead is 20% to 30%, depending the CPU affinity. No too bad, right?

Conclusion

With lua-resty-ffi, you could use your favourite mainstream programming language,
e.g. Go, Java, Python or Rust, to do development in Openresty/Nginx, so that
you could enjoy their rich ecosystem directly.

I just use 400 code lines (including lua and rust) to implement a complete GRPC client in OpenResty.
And this is just an example, you could use lua-resty-ffi to do anything you want!

Welcome to discuss and like my github repo:

https://github.com/kingluo/lua-resty-ffi!

Nginx Blocking

jinhua luo — Fri, 14 Oct 2022 16:42:26 +0000

What's blocking?

Nginx has one master process and multiple worker processes, and each worker process is single threading.

It’s well known that NGINX uses an asynchronous, event‑driven approach to handling connections. This means that instead of creating another dedicated process or thread for each request (like servers with a traditional architecture), it handles multiple connections and requests in one worker process. To achieve this, NGINX works with sockets in a non‑blocking mode and uses efficient methods such as epoll and kqueue.

However, once the connection is accepted in one worker process, it could not be moved to other processes, even
when that process is busy. Not like golang, it has no work stealing. And, only socket is asynchronous,
the cpu-intensive tasks and file io tasks is still synchronous.

Every phase handler for an http request runs in a single thread. OpenResty allows you to run lua code
in each phase. The rewrite, access, content phases, as well as the timer (implemented as faked connection)
are running lua code within an individual lua coroutine.
Within the coroutine, you could use cosocket to communicate with the world.
The cosocket is 100% nonblocking out of the box. When you call cosocket API and it needs to wait for
more data, it would yield the coroutine, register corresponding event in epoll and let the event-loop
handles other events.

As the name suggests, coroutines use the CPU in cooperative way.
When you block a long time within one coroutine, other coroutines have no way to get executed,
because all coroutines reside in the one thread, which results in uneven request handling and long request delay.

When I work in Kugou Music, my team encounters blocking issue,
and I wrote a simple tool to diagnose it. This tool
makes use of LD_PRELOAD to hook the lua_resume and check the function execution time, and
print the url and entry lua function for long execution.

In fact, this tool is not good, because it could not print the backtrace and does not work for compiled lua code.

Locate the blocking source

The best tool to check blocking should be systemtap.

The blocking comes from two things:

cpu-intensive tasks
blocking system calls

So just check the execution time, if the time exceed a threshold, print the backtraces collected during that time.

...

How to slove blocking?

Since the main thread is blocking sensitive, obviously, we should delegate blocking jobs to other threads.

The nginx provides the thread pool, which is originally used for serving static files more efficiently.
I wrap them in lua, so that you could use nonblocking API to execute blocking stuff.

I contribute my works to OpenResty official, that is ngx.run_worker_thread() API:

https://github.com/openresty/lua-nginx-module#ngxrun_worker_thread

...

Please visit my blog site for detail:
http://luajit.io/post/2022/nginx-blocking/

Analyze Lua Memory Leak With Systemtap

jinhua luo — Thu, 13 Oct 2022 14:59:56 +0000

The memory used by lua codes is managed by the GC, not calling malloc/free directly. The luajit GC uses mark-and-sweep algorithm. In simple words, it will links all gc objects in a global list. When the memory is in pressure, it would trigger recycle procedures. Note that because of single-threading design, the GC is interspersed in the flow of lua codes (either compiled
or interpretered), e.g. after string allocation. GC would try to check if the gc object is still in used, e.g. as an upvalue, reside in a stack. The GC would sweep all dangling gc objects at the end, and the corresponding memory would be freed and returned to C allocator.

In luajit, lj_alloc_malloc and lj_alloc_realloc would allocate gc objects, and lj_alloc_free would free them. So naturally, we could trace the invocations of these functions to check memory leak.

Note that triggering gc is a bit uncertain and the gc steps would be splited in different time slices. So it would not free unused memory immediately. Moreover, the lua codes may use caches. So it's necessary to check the codes to distinguish the real memory leak.

Please check my blog site:

http://luajit.io/post/2022/analyze-lua-memory-leak-with-systemtap/

Luajit String Interning

jinhua luo — Wed, 12 Oct 2022 15:35:33 +0000

Why string interning?

String Interning is a common optimization in most of programming languages.

It has below advantages in luajit:

compare string via pointers directly
use string as table key efficiently, because the hash is calculated at alloc stage and use pointer as uniqueness
save memory for duplicated strings

Drawback

Meanwhile, it brings in extra time-consuming tasks:

Hash
when collision, memcmp all strings in the same hash slot
rehash when collision is high and/or load factor is low

Analyze the overhead of string interning

How expensive is memcmp?

In a signle CPU (Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz), for each string with 1MB, only the last char is different, it could only do memcmp 90 times per second at most.

How expensive is hash collision?

We construct a workload to simulate the cache usecase (only set kv, no get).

Each http request inserts a key-value in the lua table. The key is integer and the value is string of 1MB size. Note that the key is not string, but luajit still does string interning for each string.

The http client would choose to use unique random strings or similar strings.

We use systemtap script to check the lj_str_new time cost differences.

Could we do dynamic string interning?

We could do string interning only when the string is used to:

use as table key
string comparison

Please check my blog post for detail:

http://luajit.io/post/2022/luajit-string-interning/

Gdb Black Magics

jinhua luo — Wed, 12 Oct 2022 15:07:06 +0000

Without questioin, gdb is a brilliant weapon to debug our program. However, the reality is complex. We need black magics to handle these cases.

Magic1: optimized out?

Your program is normally compiled in optimized mode (e.g. -O2), sometimes even without debug info!

With optimized option, gcc would most likely use registers instead of the stack to store the function arguments and local variables. In some circumstances, gdb would said they are , then you could not inspect their values. Even worse, in the flow of the code, those registers would be overrided, so it’s impossible to check their original values anymore!

Normally, to tackle these issues, you have to recompile your program without optimized options.

But what if you could not? Think about you’re facing your client’s production environment.

You would be frustrated, isn’t it?

Then the only way is to read the assembly codes and inspect everything you need at the right places!

Magic2: gdb script

gdb has builtin scripting. You could use it to define variables to r/w the program symbols and do some control flows upon them. It’s really useful when you want to inspect complex context infomation.

Magic3: gdb python

Besides basic scripting, if you want more magic, use python!

Python could be used to define customized command, breakpoint or function.

Magic4: set source path

Mostly it’s necessary to map the debug steps to the lines of source codes.

In fact, normally, your binary, if not stripped, would contains DWARF sections, which contains the path of the source codes. So all you need to do is to download the matching source codes and specify a correct perfix.

Please check the my blog post for detail:

http://luajit.io/post/2022/gdb-black-magics/